2024-03-25T01:39:36.186 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T01:39:36.186 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} --name yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770 -- /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/orig.config.yaml 2024-03-25T01:39:36.225 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T01:39:36.574 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7620770' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 osd op thread timeout: 150 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 108 sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObskKB0hMJpfUxtnW8i01ugVMaSIKTiGMPE9pwaWD74qdcrRNDF80UoV9ZtlOqvva7+TeHF3DSRg//B/c0wnVw= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCSw63IetpNrST60r4TDQMhJcu7ayY4c4BA+U3CukOW/GwnEviD1/7HU+6Ttzt5Qh8lFWze4ACmLVo6Q1ugJG2k= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_22:19:24 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T01:39:36.575 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa; will attempt to use it 2024-03-25T01:39:36.575 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks 2024-03-25T01:39:36.575 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T01:39:36.577 INFO:teuthology.task.internal:Checking packages... 2024-03-25T01:39:36.602 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9c429fb6abfb5deed62697d56d8997d0f0d6d83f' 2024-03-25T01:39:36.602 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T01:39:36.602 INFO:teuthology.packaging:ref: None 2024-03-25T01:39:36.602 INFO:teuthology.packaging:tag: None 2024-03-25T01:39:36.602 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-03-24-1159 2024-03-25T01:39:36.602 INFO:teuthology.packaging:sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-25T01:39:36.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri10-testing-2024-03-24-1159 2024-03-25T01:39:36.828 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2398.g9c429fb6 2024-03-25T01:39:36.829 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T01:39:36.836 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T01:39:36.836 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T01:39:36.848 INFO:teuthology.task.internal:Saving configuration 2024-03-25T01:39:36.864 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T01:39:36.870 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T01:39:36.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 01:34:33.974788', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObskKB0hMJpfUxtnW8i01ugVMaSIKTiGMPE9pwaWD74qdcrRNDF80UoV9ZtlOqvva7+TeHF3DSRg//B/c0wnVw='} 2024-03-25T01:39:36.918 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 01:34:33.972861', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCSw63IetpNrST60r4TDQMhJcu7ayY4c4BA+U3CukOW/GwnEviD1/7HU+6Ttzt5Qh8lFWze4ACmLVo6Q1ugJG2k='} 2024-03-25T01:39:36.918 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T01:39:36.924 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T01:39:36.924 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T01:39:36.924 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T01:39:36.996 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5c9a73cf70>, signals=[15]) 2024-03-25T01:39:36.997 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T01:39:37.003 INFO:teuthology.task.internal:Opening connections... 2024-03-25T01:39:37.003 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T01:39:37.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:39:37.100 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T01:39:37.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:39:37.194 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T01:39:37.200 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-03-25T01:39:37.218 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-03-25T01:39:37.219 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-03-25T01:39:37.275 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-03-25T01:39:37.276 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-03-25T01:39:37.277 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T01:39:37.277 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-03-25T01:39:37.277 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T01:39:37.277 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T01:39:37.277 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T01:39:37.278 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-03-25T01:39:37.304 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-25T01:39:37.322 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-25T01:39:37.322 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-25T01:39:37.379 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-25T01:39:37.379 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T01:39:37.380 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T01:39:37.380 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-25T01:39:37.403 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T01:39:37.412 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T01:39:37.417 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T01:39:37.417 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-03-25T01:39:37.424 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-25T01:39:37.440 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T01:39:37.446 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T01:39:37.446 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-03-25T01:39:37.480 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-25T01:39:37.502 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T01:39:37.551 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T01:39:37.565 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T01:39:37.565 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T01:39:37.566 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T01:39:37.566 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T01:39:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T01:39:37.582 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-25T01:39:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T01:39:37.583 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-25T01:39:37.638 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-361.el9.x86_64 2024-03-25T01:39:37.639 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-361.el9.x86_64 2024-03-25T01:39:37.639 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-03-25T01:39:37.640 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-03-25T01:39:37.640 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-03-25T01:39:37.640 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-03-25T01:39:39.716 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T01:39:39.965 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T01:39:41.158 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-25T01:39:41.374 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-25T01:39:42.218 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-25T01:39:44.024 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-25T01:39:48.060 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.1 MB 00:00 2024-03-25T01:39:49.358 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 96 kB/s | 16 kB 00:00 2024-03-25T01:39:50.368 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-25T01:39:53.323 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 695 kB/s | 8.0 MB 00:11 2024-03-25T01:39:55.343 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T01:39:57.475 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T01:39:57.475 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T01:39:57.565 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:39:57.576 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:39:57.576 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-25T01:39:57.576 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:39:57.576 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:39:57.576 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T01:39:57.576 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:39:57.576 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T01:39:57.577 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T01:39:57.577 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T01:39:57.577 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:39:57.577 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:39:57.577 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:39:57.577 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-03-25T01:39:57.577 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:39:57.578 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 76 M 2024-03-25T01:39:57.578 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-03-25T01:39:57.578 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:39:57.799 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-25T01:39:58.286 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 69 kB/s | 44 kB 00:00 2024-03-25T01:40:01.833 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T01:40:03.384 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T01:40:04.813 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T01:40:09.702 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-25T01:40:11.759 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T01:40:11.759 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T01:40:11.838 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:40:11.848 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:40:11.848 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-25T01:40:11.848 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-03-25T01:40:11.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:40:11.850 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 76 M 2024-03-25T01:40:11.850 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 123 M 2024-03-25T01:40:11.850 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:40:12.554 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 69 kB/s | 44 kB 00:00 2024-03-25T01:40:17.501 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 768 kB/s | 15 MB 00:19 2024-03-25T01:40:20.620 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 3.8 MB/s | 33 MB 00:08 2024-03-25T01:40:31.561 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 777 kB/s | 15 MB 00:19 2024-03-25T01:40:34.216 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 783 kB/s | 27 MB 00:35 2024-03-25T01:40:41.573 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 780 kB/s | 33 MB 00:43 2024-03-25T01:40:41.574 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:40:41.574 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.7 MB/s | 76 MB 00:43 2024-03-25T01:40:42.241 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:40:42.616 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:40:42.616 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:40:43.603 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:40:43.603 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:40:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:40:46.688 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T01:40:46.773 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T01:40:49.503 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T01:40:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T01:40:50.535 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 740 kB/s | 27 MB 00:37 2024-03-25T01:40:50.535 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:40:50.535 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.0 MB/s | 76 MB 00:38 2024-03-25T01:40:51.202 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:40:51.549 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:40:51.549 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:40:52.567 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:40:52.567 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:40:54.339 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T01:40:54.458 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:40:55.017 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:40:55.622 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T01:40:55.702 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T01:40:58.413 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T01:40:58.549 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T01:40:59.158 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:03.250 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T01:41:03.349 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:08.074 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:29.005 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:29.006 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T01:41:29.006 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:41:29.050 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:29.489 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:29.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T01:41:29.489 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T01:41:29.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:41:29.858 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:41:30.209 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T01:41:30.306 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T01:41:30.306 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T01:41:30.306 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T01:41:30.306 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T01:41:30.306 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T01:41:30.307 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T01:41:30.307 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T01:41:30.307 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T01:41:30.386 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T01:41:30.387 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T01:41:30.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-03-25T01:41:30.423 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-03-25T01:41:30.423 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-03-25T01:41:30.423 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-03-25T01:41:30.424 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:41:30.425 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T01:41:30.426 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T01:41:30.426 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-430.el9.x86_64 2024-03-25T01:41:30.426 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T01:41:31.021 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-25T01:41:32.380 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T01:41:32.401 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-25T01:41:32.403 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-25T01:41:32.418 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.l69ClaGtUL 2024-03-25T01:41:32.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.l69ClaGtUL 2024-03-25T01:41:32.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.l69ClaGtUL 2024-03-25T01:41:32.633 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.l69ClaGtUL is 6KB 2024-03-25T01:41:32.689 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.l69ClaGtUL 2024-03-25T01:41:32.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T01:41:32.771 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T01:41:32.771 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T01:41:32.772 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T01:41:32.772 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T01:41:32.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T01:41:32.930 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-25T01:41:37.491 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:37.492 INFO:teuthology.orchestra.run.smithi083.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T01:41:37.492 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:41:37.535 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:37.981 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:37.981 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T01:41:37.981 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T01:41:37.981 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T01:41:38.343 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T01:41:38.343 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:41:38.343 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:41:38.343 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T01:41:38.343 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T01:41:38.343 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T01:41:38.344 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T01:41:38.344 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:41:38.344 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:41:38.705 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T01:41:38.804 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T01:41:38.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T01:41:38.804 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T01:41:38.805 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T01:41:38.805 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T01:41:38.805 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T01:41:38.805 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T01:41:38.805 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T01:41:38.884 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T01:41:38.884 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T01:41:38.884 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-03-25T01:41:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-03-25T01:41:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-03-25T01:41:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-03-25T01:41:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 68.el9 2024-03-25T01:41:38.918 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8264254 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:41:38.919 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T01:41:38.920 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T01:41:38.920 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T01:41:38.920 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-03-25T01:41:38.920 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:41:38.920 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T01:41:38.921 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T01:41:38.921 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-430.el9.x86_64 2024-03-25T01:41:38.921 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T01:41:39.582 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-03-25T01:41:40.977 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T01:41:40.998 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-03-25T01:41:41.001 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-25T01:41:41.016 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.IxsnOFKO6m 2024-03-25T01:41:41.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IxsnOFKO6m 2024-03-25T01:41:41.084 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.IxsnOFKO6m 2024-03-25T01:41:41.225 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.IxsnOFKO6m is 6KB 2024-03-25T01:41:41.275 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.IxsnOFKO6m 2024-03-25T01:41:41.289 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T01:41:41.354 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T01:41:41.354 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T01:41:41.354 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T01:41:41.355 INFO:teuthology.orchestra.run.smithi083.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T01:41:41.356 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T01:41:41.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-03-25T01:42:02.954 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T01:42:02.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T01:42:02.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:42:11.506 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T01:42:11.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:42:11.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:42:21.474 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-25T01:42:29.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-25T01:42:30.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T01:42:30.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:42:33.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-25T01:42:38.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:42:38.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:42:41.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-25T01:42:45.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T01:42:45.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:42:53.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:42:53.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:42:57.058 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-25T01:43:03.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-25T01:43:12.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:43:12.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:43:15.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-03-25T01:43:18.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T01:43:18.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:43:19.346 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-25T01:43:19.920 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T01:43:19.920 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T01:43:19.920 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-25T01:43:19.934 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-430.el9.x86_64 2024-03-25T01:43:19.934 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T01:43:19.934 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T01:43:19.934 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T01:43:20.936 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T01:43:20.936 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T01:43:21.030 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-25T01:43:21.072 DEBUG:teuthology.parallel:result is None 2024-03-25T01:43:33.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:43:33.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:43:33.633 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-25T01:43:34.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:43:34.017 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T01:43:34.017 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-03-25T01:43:34.033 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-430.el9.x86_64 2024-03-25T01:43:34.033 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T01:43:34.033 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T01:43:34.033 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T01:43:35.034 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T01:43:35.034 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T01:43:35.104 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-03-25T01:43:35.137 DEBUG:teuthology.parallel:result is None 2024-03-25T01:43:35.137 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T01:43:35.144 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T01:43:35.144 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T01:43:35.147 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T01:43:35.165 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T01:43:35.172 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T01:43:35.181 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T01:43:35.181 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T01:43:35.205 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T01:43:35.317 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T01:43:35.323 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T01:43:35.324 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T01:43:35.326 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T01:43:35.360 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T01:43:35.363 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T01:43:35.374 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T01:43:35.377 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T01:43:35.379 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T01:43:35.386 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T01:43:35.386 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T01:43:35.422 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T01:43:35.452 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T01:43:35.462 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T01:43:35.462 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T01:43:35.492 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T01:43:35.508 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T01:43:35.587 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T01:43:35.636 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T01:43:35.718 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T01:43:35.761 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:43:35.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T01:43:35.828 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T01:43:35.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T01:43:35.927 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T01:43:36.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T01:43:36.060 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:43:36.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T01:43:36.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-03-25T01:43:36.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-25T01:43:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T01:43:36.236 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T01:43:36.687 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T01:43:36.695 INFO:teuthology.task.internal:Starting timer... 2024-03-25T01:43:36.695 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T01:43:36.705 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T01:43:36.715 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-25T01:43:36.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-03-25T01:43:36.789 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-03-25T01:43:36.791 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-25T01:43:36.857 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-25T01:43:36.859 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T01:43:36.859 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-03-25T01:43:36.888 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-03-25T01:43:36.888 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-25T01:43:36.914 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-25T01:43:36.914 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-03-25T01:43:36.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-25T01:43:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T01:43:36.957 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-25T01:43:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T01:43:36.987 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T01:43:36.987 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-03-25T01:43:37.032 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-25T01:43:37.065 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T01:43:37.076 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T01:43:37.094 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T01:43:37.095 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T01:47:50.019 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-03-25T01:47:50.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:47:50.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:47:50.117 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-25T01:47:50.197 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-25T01:47:50.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T01:47:50.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T01:47:50.318 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-25T01:47:50.399 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-25T01:47:50.400 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T01:47:50.410 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T01:47:50.410 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T01:47:50.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T01:47:50.413 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T01:47:50.413 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T01:47:50.442 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T01:47:50.459 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T01:47:50.473 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T01:47:50.488 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-03-25T01:47:50.491 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T01:47:50.501 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-03-25T01:47:50.518 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T01:47:50.521 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-25T01:47:50.534 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T01:47:50.535 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-25T01:47:50.552 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T01:47:50.568 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T01:47:50.591 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-25T01:47:50.595 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T01:47:50.595 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-25T01:47:50.596 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.596 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.596 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.596 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.619 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.623 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T01:47:50.624 INFO:teuthology.run_tasks:Running task install... 2024-03-25T01:47:50.633 DEBUG:teuthology.task.install:project ceph 2024-03-25T01:47:50.633 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'}} 2024-03-25T01:47:50.633 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'} 2024-03-25T01:47:50.633 INFO:teuthology.task.install:Using flavor: default 2024-03-25T01:47:50.640 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T01:47:50.640 INFO:teuthology.task.install:extra packages: [] 2024-03-25T01:47:50.641 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f', 'tag': None, 'wait_for_package': False} 2024-03-25T01:47:50.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-25T01:47:50.643 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f', 'tag': None, 'wait_for_package': False} 2024-03-25T01:47:50.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-25T01:47:50.791 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/ 2024-03-25T01:47:50.791 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2398.g9c429fb6 2024-03-25T01:47:50.863 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/ 2024-03-25T01:47:50.863 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2398.g9c429fb6 2024-03-25T01:47:50.903 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T01:47:50.903 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:47:50.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T01:47:50.938 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T01:47:50.938 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T01:47:51.006 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T01:47:51.038 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T01:47:51.039 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:47:51.039 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T01:47:51.067 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T01:47:51.067 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T01:47:51.086 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T01:47:51.135 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T01:47:51.152 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-25T01:47:51.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-25T01:47:51.215 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T01:47:51.281 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-03-25T01:47:51.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-03-25T01:47:51.407 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-25T01:47:51.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-03-25T01:47:51.525 INFO:teuthology.orchestra.run.smithi083.stdout:74 files removed 2024-03-25T01:47:51.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-03-25T01:47:52.155 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 273 kB/s | 83 kB 00:00 2024-03-25T01:47:52.202 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-03-25T01:47:52.522 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-03-25T01:47:52.586 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-03-25T01:47:52.912 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-25T01:47:53.048 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-25T01:47:53.431 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T01:47:53.517 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T01:47:55.357 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 2.5 MB/s | 2.0 MB 00:00 2024-03-25T01:47:55.359 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 2.7 MB/s | 2.0 MB 00:00 2024-03-25T01:47:56.564 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-03-25T01:47:58.365 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-25T01:47:58.525 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.0 MB 00:02 2024-03-25T01:48:04.776 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 19 MB 00:05 2024-03-25T01:48:10.448 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 728 kB/s | 6.1 MB 00:08 2024-03-25T01:48:10.843 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-25T01:48:12.207 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-25T01:48:12.802 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-25T01:48:13.158 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-25T01:48:14.431 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T01:48:18.050 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T01:48:19.337 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-25T01:48:20.487 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:48:20.506 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:48:20.506 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:48:20.506 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:48:20.506 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:48:20.506 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 9.8 M 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 3.4 M 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 3.1 M 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 5.4 M 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 21 M 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 25 k 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T01:48:20.507 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 746 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 501 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 5.1 M 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 45 k 2024-03-25T01:48:20.508 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 133 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 162 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 321 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 298 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 99 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:48:20.509 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:48:20.510 INFO:teuthology.orchestra.run.smithi153.stdout:Install 30 Packages 2024-03-25T01:48:20.510 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-03-25T01:48:20.510 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:20.510 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 59 M 2024-03-25T01:48:20.510 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:48:21.774 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:48:21.794 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:48:21.794 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:48:21.794 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:48:21.794 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:48:21.794 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 9.8 M 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 3.4 M 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 3.1 M 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 5.4 M 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 21 M 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 25 k 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T01:48:21.795 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 746 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 501 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 5.1 M 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 45 k 2024-03-25T01:48:21.796 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 133 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 162 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 321 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 298 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 99 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:48:21.797 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:48:21.798 INFO:teuthology.orchestra.run.smithi083.stdout:Install 30 Packages 2024-03-25T01:48:21.798 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-03-25T01:48:21.798 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:21.798 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 59 M 2024-03-25T01:48:21.798 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:48:21.968 INFO:teuthology.orchestra.run.smithi153.stdout:(1/32): ceph-base-19.0.0-2398.g9c429fb6.el9.x86 4.0 MB/s | 5.4 MB 00:01 2024-03-25T01:48:22.176 INFO:teuthology.orchestra.run.smithi153.stdout:(2/32): ceph-common-19.0.0-2398.g9c429fb6.el9.x 13 MB/s | 21 MB 00:01 2024-03-25T01:48:22.201 INFO:teuthology.orchestra.run.smithi153.stdout:(3/32): ceph-selinux-19.0.0-2398.g9c429fb6.el9. 108 kB/s | 25 kB 00:00 2024-03-25T01:48:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:(4/32): libcephfs2-19.0.0-2398.g9c429fb6.el9.x8 9.7 MB/s | 746 kB 00:00 2024-03-25T01:48:22.311 INFO:teuthology.orchestra.run.smithi153.stdout:(5/32): libradosstriper1-19.0.0-2398.g9c429fb6. 4.5 MB/s | 501 kB 00:00 2024-03-25T01:48:22.403 INFO:teuthology.orchestra.run.smithi153.stdout:(6/32): python3-ceph-argparse-19.0.0-2398.g9c42 492 kB/s | 45 kB 00:00 2024-03-25T01:48:22.503 INFO:teuthology.orchestra.run.smithi153.stdout:(7/32): python3-ceph-common-19.0.0-2398.g9c429f 1.3 MB/s | 133 kB 00:00 2024-03-25T01:48:22.628 INFO:teuthology.orchestra.run.smithi153.stdout:(8/32): librgw2-19.0.0-2398.g9c429fb6.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-25T01:48:22.653 INFO:teuthology.orchestra.run.smithi153.stdout:(9/32): python3-cephfs-19.0.0-2398.g9c429fb6.el 1.1 MB/s | 162 kB 00:00 2024-03-25T01:48:22.687 INFO:teuthology.orchestra.run.smithi153.stdout:(10/32): python3-rados-19.0.0-2398.g9c429fb6.el 5.4 MB/s | 321 kB 00:00 2024-03-25T01:48:22.738 INFO:teuthology.orchestra.run.smithi153.stdout:(11/32): python3-rbd-19.0.0-2398.g9c429fb6.el9. 3.5 MB/s | 298 kB 00:00 2024-03-25T01:48:22.782 INFO:teuthology.orchestra.run.smithi153.stdout:(12/32): python3-rgw-19.0.0-2398.g9c429fb6.el9. 1.0 MB/s | 99 kB 00:00 2024-03-25T01:48:22.888 INFO:teuthology.orchestra.run.smithi083.stdout:(1/32): ceph-base-19.0.0-2398.g9c429fb6.el9.x86 6.1 MB/s | 5.4 MB 00:00 2024-03-25T01:48:22.905 INFO:teuthology.orchestra.run.smithi153.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 634 kB/s | 106 kB 00:00 2024-03-25T01:48:22.921 INFO:teuthology.orchestra.run.smithi083.stdout:(2/32): ceph-selinux-19.0.0-2398.g9c429fb6.el9. 763 kB/s | 25 kB 00:00 2024-03-25T01:48:22.930 INFO:teuthology.orchestra.run.smithi153.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-25T01:48:22.947 INFO:teuthology.orchestra.run.smithi153.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T01:48:22.997 INFO:teuthology.orchestra.run.smithi083.stdout:(3/32): libcephfs2-19.0.0-2398.g9c429fb6.el9.x8 9.7 MB/s | 746 kB 00:00 2024-03-25T01:48:23.047 INFO:teuthology.orchestra.run.smithi153.stdout:(16/32): ceph-radosgw-19.0.0-2398.g9c429fb6.el9 4.0 MB/s | 9.8 MB 00:02 2024-03-25T01:48:23.064 INFO:teuthology.orchestra.run.smithi083.stdout:(4/32): libradosstriper1-19.0.0-2398.g9c429fb6. 7.3 MB/s | 501 kB 00:00 2024-03-25T01:48:23.098 INFO:teuthology.orchestra.run.smithi153.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 661 kB 00:00 2024-03-25T01:48:23.123 INFO:teuthology.orchestra.run.smithi153.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 282 kB 00:00 2024-03-25T01:48:23.148 INFO:teuthology.orchestra.run.smithi153.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 413 kB/s | 41 kB 00:00 2024-03-25T01:48:23.257 INFO:teuthology.orchestra.run.smithi153.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-03-25T01:48:23.282 INFO:teuthology.orchestra.run.smithi153.stdout:(21/32): qatlib-service-23.11.0-1.el9.x86_64.rp 261 kB/s | 35 kB 00:00 2024-03-25T01:48:23.342 INFO:teuthology.orchestra.run.smithi153.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 895 kB/s | 216 kB 00:00 2024-03-25T01:48:23.381 INFO:teuthology.orchestra.run.smithi083.stdout:(5/32): librgw2-19.0.0-2398.g9c429fb6.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-25T01:48:23.422 INFO:teuthology.orchestra.run.smithi083.stdout:(6/32): python3-ceph-argparse-19.0.0-2398.g9c42 1.1 MB/s | 45 kB 00:00 2024-03-25T01:48:23.424 INFO:teuthology.orchestra.run.smithi153.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-25T01:48:23.450 INFO:teuthology.orchestra.run.smithi153.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-25T01:48:23.473 INFO:teuthology.orchestra.run.smithi083.stdout:(7/32): python3-ceph-common-19.0.0-2398.g9c429f 2.6 MB/s | 133 kB 00:00 2024-03-25T01:48:23.530 INFO:teuthology.orchestra.run.smithi153.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 725 kB/s | 49 kB 00:00 2024-03-25T01:48:23.530 INFO:teuthology.orchestra.run.smithi153.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T01:48:23.531 INFO:teuthology.orchestra.run.smithi083.stdout:(8/32): python3-cephfs-19.0.0-2398.g9c429fb6.el 2.7 MB/s | 162 kB 00:00 2024-03-25T01:48:23.590 INFO:teuthology.orchestra.run.smithi083.stdout:(9/32): python3-rados-19.0.0-2398.g9c429fb6.el9 5.4 MB/s | 321 kB 00:00 2024-03-25T01:48:23.609 INFO:teuthology.orchestra.run.smithi153.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-25T01:48:23.651 INFO:teuthology.orchestra.run.smithi153.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-25T01:48:23.657 INFO:teuthology.orchestra.run.smithi083.stdout:(10/32): python3-rbd-19.0.0-2398.g9c429fb6.el9. 4.4 MB/s | 298 kB 00:00 2024-03-25T01:48:23.676 INFO:teuthology.orchestra.run.smithi153.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-25T01:48:23.724 INFO:teuthology.orchestra.run.smithi083.stdout:(11/32): python3-rgw-19.0.0-2398.g9c429fb6.el9. 1.5 MB/s | 99 kB 00:00 2024-03-25T01:48:23.726 INFO:teuthology.orchestra.run.smithi153.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-25T01:48:23.824 INFO:teuthology.orchestra.run.smithi083.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-25T01:48:23.866 INFO:teuthology.orchestra.run.smithi083.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T01:48:23.900 INFO:teuthology.orchestra.run.smithi083.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T01:48:23.950 INFO:teuthology.orchestra.run.smithi083.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-25T01:48:23.992 INFO:teuthology.orchestra.run.smithi083.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-25T01:48:24.034 INFO:teuthology.orchestra.run.smithi083.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 985 kB/s | 41 kB 00:00 2024-03-25T01:48:24.177 INFO:teuthology.orchestra.run.smithi153.stdout:(31/32): librbd1-19.0.0-2398.g9c429fb6.el9.x86_ 6.3 MB/s | 3.1 MB 00:00 2024-03-25T01:48:24.242 INFO:teuthology.orchestra.run.smithi083.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T01:48:24.293 INFO:teuthology.orchestra.run.smithi083.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-03-25T01:48:24.343 INFO:teuthology.orchestra.run.smithi083.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 895 kB/s | 44 kB 00:00 2024-03-25T01:48:24.510 INFO:teuthology.orchestra.run.smithi153.stdout:(32/32): librados2-19.0.0-2398.g9c429fb6.el9.x8 4.0 MB/s | 3.4 MB 00:00 2024-03-25T01:48:24.514 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:48:24.514 INFO:teuthology.orchestra.run.smithi153.stdout:Total 15 MB/s | 59 MB 00:04 2024-03-25T01:48:24.549 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:48:24.588 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:48:24.588 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:48:24.860 INFO:teuthology.orchestra.run.smithi083.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 598 kB/s | 309 kB 00:00 2024-03-25T01:48:24.977 INFO:teuthology.orchestra.run.smithi083.stdout:(22/32): ceph-radosgw-19.0.0-2398.g9c429fb6.el9 3.3 MB/s | 9.8 MB 00:02 2024-03-25T01:48:25.085 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:48:25.085 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:48:25.219 INFO:teuthology.orchestra.run.smithi083.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 107 kB/s | 26 kB 00:00 2024-03-25T01:48:25.295 INFO:teuthology.orchestra.run.smithi083.stdout:(24/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-25T01:48:25.503 INFO:teuthology.orchestra.run.smithi083.stdout:(25/32): ceph-common-19.0.0-2398.g9c429fb6.el9. 5.9 MB/s | 21 MB 00:03 2024-03-25T01:48:25.520 INFO:teuthology.orchestra.run.smithi083.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 161 kB/s | 49 kB 00:00 2024-03-25T01:48:25.545 INFO:teuthology.orchestra.run.smithi083.stdout:(27/32): libunwind-1.6.2-1.el9.x86_64.rpm 269 kB/s | 67 kB 00:00 2024-03-25T01:48:25.687 INFO:teuthology.orchestra.run.smithi083.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-25T01:48:25.804 INFO:teuthology.orchestra.run.smithi083.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 675 kB/s | 192 kB 00:00 2024-03-25T01:48:26.080 INFO:teuthology.orchestra.run.smithi083.stdout:(30/32): librados2-19.0.0-2398.g9c429fb6.el9.x8 8.7 MB/s | 3.4 MB 00:00 2024-03-25T01:48:26.146 INFO:teuthology.orchestra.run.smithi083.stdout:(31/32): librbd1-19.0.0-2398.g9c429fb6.el9.x86_ 9.2 MB/s | 3.1 MB 00:00 2024-03-25T01:48:26.188 INFO:teuthology.orchestra.run.smithi083.stdout:(32/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-25T01:48:26.192 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:48:26.192 INFO:teuthology.orchestra.run.smithi083.stdout:Total 13 MB/s | 59 MB 00:04 2024-03-25T01:48:26.226 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:48:26.264 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:48:26.264 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:48:26.565 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:48:26.658 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T01:48:26.698 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T01:48:26.724 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T01:48:26.746 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T01:48:26.754 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:48:26.755 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:48:26.763 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T01:48:26.916 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T01:48:26.977 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T01:48:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T01:48:27.392 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 7/34 2024-03-25T01:48:27.484 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 7/34 2024-03-25T01:48:27.580 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 8/34 2024-03-25T01:48:27.610 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 9/34 2024-03-25T01:48:27.673 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 9/34 2024-03-25T01:48:27.702 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T01:48:27.745 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2398.g9c429fb6.el9. 11/34 2024-03-25T01:48:27.814 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 12/34 2024-03-25T01:48:27.838 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_6 13/34 2024-03-25T01:48:27.894 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_6 13/34 2024-03-25T01:48:27.931 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T01:48:28.098 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T01:48:28.133 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T01:48:28.183 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T01:48:28.212 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T01:48:28.229 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:48:28.243 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T01:48:28.275 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T01:48:28.308 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T01:48:28.343 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T01:48:28.376 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T01:48:28.400 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T01:48:28.416 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T01:48:28.616 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T01:48:28.623 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T01:48:28.652 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 22/34 2024-03-25T01:48:28.671 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T01:48:28.743 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 22/34 2024-03-25T01:48:28.774 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 23/34 2024-03-25T01:48:28.820 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T01:48:28.992 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T01:48:29.045 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 7/34 2024-03-25T01:48:29.120 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 7/34 2024-03-25T01:48:29.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T01:48:29.305 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 26/34 2024-03-25T01:48:29.331 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 8/34 2024-03-25T01:48:29.355 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 9/34 2024-03-25T01:48:29.368 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 26/34 2024-03-25T01:48:29.401 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 9/34 2024-03-25T01:48:29.430 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T01:48:29.445 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 27/34 2024-03-25T01:48:29.474 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:19.0.0-2398.g9c429fb6.el9. 11/34 2024-03-25T01:48:29.474 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2398.g9c429fb6.el9.x8 28/34 2024-03-25T01:48:29.543 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 12/34 2024-03-25T01:48:29.565 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_6 13/34 2024-03-25T01:48:29.628 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_6 13/34 2024-03-25T01:48:29.656 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T01:48:29.766 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T01:48:29.798 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T01:48:29.838 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T01:48:29.871 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T01:48:29.900 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T01:48:29.941 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T01:48:30.260 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T01:48:30.296 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 22/34 2024-03-25T01:48:30.370 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 22/34 2024-03-25T01:48:30.401 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 23/34 2024-03-25T01:48:30.439 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T01:48:30.908 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T01:48:30.941 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 26/34 2024-03-25T01:48:31.012 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 26/34 2024-03-25T01:48:31.079 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 27/34 2024-03-25T01:48:31.110 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:19.0.0-2398.g9c429fb6.el9.x8 28/34 2024-03-25T01:48:31.752 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 29/34 2024-03-25T01:48:31.804 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 29/34 2024-03-25T01:48:32.591 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 29/34 2024-03-25T01:48:32.641 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 30/34 2024-03-25T01:48:32.716 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 30/34 2024-03-25T01:48:32.716 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T01:48:32.717 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T01:48:32.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:32.776 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 31/34 2024-03-25T01:48:33.379 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 29/34 2024-03-25T01:48:33.423 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 29/34 2024-03-25T01:48:33.991 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 29/34 2024-03-25T01:48:34.026 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 30/34 2024-03-25T01:48:34.108 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 30/34 2024-03-25T01:48:34.108 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T01:48:34.108 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T01:48:34.108 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:34.144 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 31/34 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 31/34 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-25T01:48:42.257 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:43.199 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/34 2024-03-25T01:48:43.225 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/34 2024-03-25T01:48:43.226 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:48:43.226 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T01:48:43.226 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T01:48:43.226 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T01:48:43.226 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:43.227 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T01:48:43.313 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T01:48:43.313 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T01:48:43.841 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 31/34 2024-03-25T01:48:43.841 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-03-25T01:48:43.841 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-03-25T01:48:43.841 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-03-25T01:48:43.841 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-03-25T01:48:43.841 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-03-25T01:48:43.842 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-03-25T01:48:43.842 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-03-25T01:48:43.842 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 3/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 4/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_6 6/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 7/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2398.g9c429fb6.el9. 8/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2398.g9c429fb6.el9.x8 9/34 2024-03-25T01:48:44.465 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 10/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 11/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 12/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 13/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T01:48:44.466 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 31/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T01:48:44.467 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 33/34 2024-03-25T01:48:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/34 2024-03-25T01:48:44.747 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/34 2024-03-25T01:48:44.747 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:48:44.747 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T01:48:44.747 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T01:48:44.747 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T01:48:44.747 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:44.749 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T01:48:44.836 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T01:48:44.836 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T01:48:45.490 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T01:48:45.491 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.492 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:45.493 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:48:45.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-03-25T01:48:46.007 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T01:48:46.007 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/34 2024-03-25T01:48:46.007 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 3/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 4/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_6 6/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 7/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2398.g9c429fb6.el9. 8/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:19.0.0-2398.g9c429fb6.el9.x8 9/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 10/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 11/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 12/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 13/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T01:48:46.008 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T01:48:46.009 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T01:48:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T01:48:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T01:48:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T01:48:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 31/34 2024-03-25T01:48:46.010 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T01:48:46.011 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 33/34 2024-03-25T01:48:46.366 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:28 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:48:46.524 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 48 M 2024-03-25T01:48:46.525 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 163 k 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-25T01:48:46.526 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:46.527 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 49 M 2024-03-25T01:48:46.527 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 206 M 2024-03-25T01:48:46.527 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:48:46.861 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): libcephsqlite-19.0.0-2398.g9c429fb6.el9. 604 kB/s | 163 kB 00:00 2024-03-25T01:48:46.928 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 643 kB/s | 217 kB 00:00 2024-03-25T01:48:47.019 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2024-03-25T01:48:47.036 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 578 kB/s | 63 kB 00:00 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.168 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T01:48:47.169 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:47.170 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:48:47.171 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T01:48:47.489 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-03-25T01:48:48.041 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:29 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:48:48.198 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 48 M 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T01:48:48.199 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 163 k 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-03-25T01:48:48.200 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:48.201 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 49 M 2024-03-25T01:48:48.201 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 206 M 2024-03-25T01:48:48.201 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:48:48.929 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 586 kB/s | 217 kB 00:00 2024-03-25T01:48:48.954 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): libcephsqlite-19.0.0-2398.g9c429fb6.el9. 413 kB/s | 163 kB 00:00 2024-03-25T01:48:49.055 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-25T01:48:49.080 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 500 kB/s | 63 kB 00:00 2024-03-25T01:48:49.220 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-19.0.0-2398.g9c429fb6.el9.x86_ 18 MB/s | 48 MB 00:02 2024-03-25T01:48:49.223 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:48:49.223 INFO:teuthology.orchestra.run.smithi153.stdout:Total 18 MB/s | 49 MB 00:02 2024-03-25T01:48:49.238 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:48:49.259 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:48:49.259 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:48:49.646 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:48:49.646 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:48:49.772 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 261 kB/s | 187 kB 00:00 2024-03-25T01:48:50.221 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:48:50.296 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T01:48:50.330 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T01:48:50.361 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T01:48:50.408 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T01:48:50.440 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/6 2024-03-25T01:48:51.314 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): ceph-test-19.0.0-2398.g9c429fb6.el9.x86_ 17 MB/s | 48 MB 00:02 2024-03-25T01:48:51.317 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:48:51.317 INFO:teuthology.orchestra.run.smithi083.stdout:Total 16 MB/s | 49 MB 00:03 2024-03-25T01:48:51.331 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:48:51.353 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:48:51.354 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:48:51.734 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:48:51.735 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:48:52.312 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:48:52.382 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T01:48:52.415 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T01:48:52.480 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T01:48:52.529 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T01:48:52.559 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/6 2024-03-25T01:48:54.955 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/6 2024-03-25T01:48:55.030 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 6/6 2024-03-25T01:48:57.092 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/6 2024-03-25T01:48:57.167 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 6/6 2024-03-25T01:48:57.499 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 6/6 2024-03-25T01:48:57.499 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/6 2024-03-25T01:48:57.499 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/6 2024-03-25T01:48:57.499 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T01:48:57.500 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T01:48:57.500 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T01:48:57.908 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T01:48:57.908 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:57.908 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:48:57.908 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:57.908 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-25T01:48:57.908 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:57.908 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T01:48:57.909 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T01:48:57.909 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T01:48:57.909 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:57.909 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:48:58.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-03-25T01:48:58.804 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:40 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:48:58.959 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:48:58.961 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-25T01:48:58.961 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 6.6 k 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 2.4 M 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.5 M 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 248 k 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 4.7 M 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 17 M 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T01:48:58.962 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T01:48:58.963 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T01:48:58.964 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T01:48:58.965 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T01:48:58.965 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:58.965 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:48:58.965 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-03-25T01:48:58.965 INFO:teuthology.orchestra.run.smithi153.stdout:Install 36 Packages 2024-03-25T01:48:58.965 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:48:58.965 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 30 M 2024-03-25T01:48:58.966 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 108 M 2024-03-25T01:48:58.966 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:48:59.515 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 6/6 2024-03-25T01:48:59.515 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/6 2024-03-25T01:48:59.516 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/6 2024-03-25T01:48:59.516 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T01:48:59.516 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T01:48:59.516 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T01:48:59.706 INFO:teuthology.orchestra.run.smithi153.stdout:(1/36): ceph-19.0.0-2398.g9c429fb6.el9.x86_64.r 33 kB/s | 6.6 kB 00:00 2024-03-25T01:48:59.848 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T01:48:59.848 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:59.848 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:48:59.848 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:59.848 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-16.el9.x86_64 2024-03-25T01:48:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:48:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T01:48:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T01:48:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T01:48:59.849 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:48:59.849 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:00.073 INFO:teuthology.orchestra.run.smithi153.stdout:(2/36): ceph-mon-19.0.0-2398.g9c429fb6.el9.x86_ 13 MB/s | 4.7 MB 00:00 2024-03-25T01:49:00.119 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-03-25T01:49:00.324 INFO:teuthology.orchestra.run.smithi153.stdout:(3/36): ceph-mgr-19.0.0-2398.g9c429fb6.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2024-03-25T01:49:00.432 INFO:teuthology.orchestra.run.smithi153.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2398.g9c42 2.2 MB/s | 248 kB 00:00 2024-03-25T01:49:00.532 INFO:teuthology.orchestra.run.smithi153.stdout:(5/36): ceph-mds-19.0.0-2398.g9c429fb6.el9.x86_ 2.3 MB/s | 2.4 MB 00:01 2024-03-25T01:49:00.616 INFO:teuthology.orchestra.run.smithi153.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-25T01:49:00.641 INFO:teuthology.orchestra.run.smithi153.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 895 kB/s | 187 kB 00:00 2024-03-25T01:49:00.666 INFO:teuthology.orchestra.run.smithi153.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 636 kB/s | 32 kB 00:00 2024-03-25T01:49:00.682 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:41 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:00.692 INFO:teuthology.orchestra.run.smithi153.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-25T01:49:00.717 INFO:teuthology.orchestra.run.smithi153.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-25T01:49:00.742 INFO:teuthology.orchestra.run.smithi153.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T01:49:00.801 INFO:teuthology.orchestra.run.smithi153.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T01:49:00.826 INFO:teuthology.orchestra.run.smithi153.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-25T01:49:00.832 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:00.834 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-25T01:49:00.834 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:00.834 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-25T01:49:00.834 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:00.834 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 6.6 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 2.4 M 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.5 M 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 248 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 4.7 M 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 17 M 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T01:49:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T01:49:00.836 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout:========================================================================================= 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout:Install 36 Packages 2024-03-25T01:49:00.837 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:00.838 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 30 M 2024-03-25T01:49:00.838 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 108 M 2024-03-25T01:49:00.838 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:00.852 INFO:teuthology.orchestra.run.smithi153.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T01:49:00.877 INFO:teuthology.orchestra.run.smithi153.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T01:49:01.036 INFO:teuthology.orchestra.run.smithi153.stdout:(16/36): ceph-osd-19.0.0-2398.g9c429fb6.el9.x86 18 MB/s | 17 MB 00:00 2024-03-25T01:49:01.102 INFO:teuthology.orchestra.run.smithi153.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-25T01:49:01.236 INFO:teuthology.orchestra.run.smithi153.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 329 kB/s | 43 kB 00:00 2024-03-25T01:49:01.278 INFO:teuthology.orchestra.run.smithi153.stdout:(19/36): python3-requests-2.25.1-8.el9.noarch.r 293 kB/s | 125 kB 00:00 2024-03-25T01:49:01.293 INFO:teuthology.orchestra.run.smithi083.stdout:(1/36): ceph-19.0.0-2398.g9c429fb6.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-25T01:49:01.303 INFO:teuthology.orchestra.run.smithi153.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 620 kB/s | 151 kB 00:00 2024-03-25T01:49:01.379 INFO:teuthology.orchestra.run.smithi153.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T01:49:01.395 INFO:teuthology.orchestra.run.smithi153.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-25T01:49:01.420 INFO:teuthology.orchestra.run.smithi153.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-03-25T01:49:01.446 INFO:teuthology.orchestra.run.smithi153.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 263 kB/s | 18 kB 00:00 2024-03-25T01:49:01.471 INFO:teuthology.orchestra.run.smithi153.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 306 kB/s | 23 kB 00:00 2024-03-25T01:49:01.496 INFO:teuthology.orchestra.run.smithi153.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-25T01:49:01.518 INFO:teuthology.orchestra.run.smithi083.stdout:(2/36): ceph-mds-19.0.0-2398.g9c429fb6.el9.x86_ 6.0 MB/s | 2.4 MB 00:00 2024-03-25T01:49:01.521 INFO:teuthology.orchestra.run.smithi153.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-25T01:49:01.563 INFO:teuthology.orchestra.run.smithi153.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-25T01:49:01.589 INFO:teuthology.orchestra.run.smithi153.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 854 kB/s | 79 kB 00:00 2024-03-25T01:49:01.622 INFO:teuthology.orchestra.run.smithi153.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2024-03-25T01:49:01.648 INFO:teuthology.orchestra.run.smithi153.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T01:49:01.673 INFO:teuthology.orchestra.run.smithi153.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-25T01:49:01.698 INFO:teuthology.orchestra.run.smithi153.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 473 kB/s | 36 kB 00:00 2024-03-25T01:49:01.749 INFO:teuthology.orchestra.run.smithi153.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T01:49:01.774 INFO:teuthology.orchestra.run.smithi153.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-25T01:49:01.799 INFO:teuthology.orchestra.run.smithi153.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-25T01:49:01.802 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:01.802 INFO:teuthology.orchestra.run.smithi153.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-25T01:49:01.810 INFO:teuthology.orchestra.run.smithi083.stdout:(3/36): ceph-mgr-19.0.0-2398.g9c429fb6.el9.x86_ 2.2 MB/s | 1.5 MB 00:00 2024-03-25T01:49:01.824 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:01.866 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:01.866 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:01.868 INFO:teuthology.orchestra.run.smithi083.stdout:(4/36): ceph-mon-19.0.0-2398.g9c429fb6.el9.x86_ 8.2 MB/s | 4.7 MB 00:00 2024-03-25T01:49:01.894 INFO:teuthology.orchestra.run.smithi083.stdout:(5/36): ceph-mgr-modules-core-19.0.0-2398.g9c42 2.9 MB/s | 248 kB 00:00 2024-03-25T01:49:02.028 INFO:teuthology.orchestra.run.smithi083.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-25T01:49:02.054 INFO:teuthology.orchestra.run.smithi083.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-03-25T01:49:02.069 INFO:teuthology.orchestra.run.smithi083.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-03-25T01:49:02.086 INFO:teuthology.orchestra.run.smithi083.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T01:49:02.103 INFO:teuthology.orchestra.run.smithi083.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T01:49:02.154 INFO:teuthology.orchestra.run.smithi083.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-25T01:49:02.262 INFO:teuthology.orchestra.run.smithi083.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 7.2 MB/s | 1.1 MB 00:00 2024-03-25T01:49:02.287 INFO:teuthology.orchestra.run.smithi083.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 773 kB/s | 103 kB 00:00 2024-03-25T01:49:02.294 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:02.294 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:02.313 INFO:teuthology.orchestra.run.smithi083.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T01:49:02.347 INFO:teuthology.orchestra.run.smithi083.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-25T01:49:02.430 INFO:teuthology.orchestra.run.smithi083.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-25T01:49:02.614 INFO:teuthology.orchestra.run.smithi083.stdout:(17/36): ceph-osd-19.0.0-2398.g9c429fb6.el9.x86 15 MB/s | 17 MB 00:01 2024-03-25T01:49:02.964 INFO:teuthology.orchestra.run.smithi083.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 192 kB/s | 125 kB 00:00 2024-03-25T01:49:03.256 INFO:teuthology.orchestra.run.smithi083.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 183 kB/s | 151 kB 00:00 2024-03-25T01:49:03.282 INFO:teuthology.orchestra.run.smithi083.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 67 kB/s | 43 kB 00:00 2024-03-25T01:49:03.398 INFO:teuthology.orchestra.run.smithi083.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-03-25T01:49:03.515 INFO:teuthology.orchestra.run.smithi083.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 152 kB/s | 18 kB 00:00 2024-03-25T01:49:03.632 INFO:teuthology.orchestra.run.smithi083.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-25T01:49:03.662 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:03.727 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T01:49:03.732 INFO:teuthology.orchestra.run.smithi083.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T01:49:03.767 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T01:49:03.783 INFO:teuthology.orchestra.run.smithi083.stdout:(25/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 210 kB/s | 172 kB 00:00 2024-03-25T01:49:03.796 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T01:49:03.825 INFO:teuthology.orchestra.run.smithi083.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T01:49:03.850 INFO:teuthology.orchestra.run.smithi083.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 603 kB/s | 358 kB 00:00 2024-03-25T01:49:03.865 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T01:49:03.875 INFO:teuthology.orchestra.run.smithi083.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-25T01:49:03.903 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T01:49:03.935 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T01:49:03.976 INFO:teuthology.orchestra.run.smithi083.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T01:49:04.001 INFO:teuthology.orchestra.run.smithi083.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 448 kB/s | 79 kB 00:00 2024-03-25T01:49:04.028 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T01:49:04.093 INFO:teuthology.orchestra.run.smithi083.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-03-25T01:49:04.115 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T01:49:04.152 INFO:teuthology.orchestra.run.smithi083.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 512 kB/s | 90 kB 00:00 2024-03-25T01:49:04.155 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T01:49:04.202 INFO:teuthology.orchestra.run.smithi083.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 775 kB/s | 272 kB 00:00 2024-03-25T01:49:04.226 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T01:49:04.289 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T01:49:04.294 INFO:teuthology.orchestra.run.smithi083.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-25T01:49:04.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T01:49:04.377 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T01:49:04.441 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T01:49:04.635 INFO:teuthology.orchestra.run.smithi083.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 654 kB/s | 230 kB 00:00 2024-03-25T01:49:04.635 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T01:49:04.682 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T01:49:04.720 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T01:49:04.755 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T01:49:04.782 INFO:teuthology.orchestra.run.smithi083.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 678 kB/s | 427 kB 00:00 2024-03-25T01:49:04.789 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:04.789 INFO:teuthology.orchestra.run.smithi083.stdout:Total 7.6 MB/s | 30 MB 00:03 2024-03-25T01:49:04.793 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T01:49:04.814 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:04.857 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:04.857 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:04.878 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T01:49:04.912 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T01:49:04.954 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T01:49:04.994 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T01:49:05.029 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T01:49:05.064 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T01:49:05.152 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T01:49:05.194 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T01:49:05.266 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T01:49:05.272 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:05.273 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:05.412 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T01:49:05.554 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T01:49:05.720 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2398.g9c429fb6.el9. 31/36 2024-03-25T01:49:05.758 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/36 2024-03-25T01:49:05.783 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/36 2024-03-25T01:49:05.783 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:05.783 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T01:49:05.783 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T01:49:05.784 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T01:49:05.784 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:06.602 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:06.664 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T01:49:06.696 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T01:49:06.724 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T01:49:06.796 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T01:49:06.857 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T01:49:06.889 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T01:49:06.981 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T01:49:07.051 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T01:49:07.093 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T01:49:07.164 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T01:49:07.217 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 33/36 2024-03-25T01:49:07.218 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T01:49:07.243 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 33/36 2024-03-25T01:49:07.243 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:07.243 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T01:49:07.243 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T01:49:07.243 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T01:49:07.243 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:07.257 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T01:49:07.305 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T01:49:07.361 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T01:49:07.545 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T01:49:07.594 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T01:49:07.634 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T01:49:07.668 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T01:49:07.676 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 34/36 2024-03-25T01:49:07.698 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T01:49:07.698 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 34/36 2024-03-25T01:49:07.698 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:07.699 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T01:49:07.699 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T01:49:07.699 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T01:49:07.699 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:07.774 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T01:49:07.817 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T01:49:07.850 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T01:49:07.890 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T01:49:07.925 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T01:49:07.926 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 35/36 2024-03-25T01:49:07.949 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 35/36 2024-03-25T01:49:07.949 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:07.949 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T01:49:07.950 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T01:49:07.950 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T01:49:07.950 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:07.960 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T01:49:08.015 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 36/36 2024-03-25T01:49:08.052 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T01:49:08.092 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T01:49:08.163 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T01:49:08.303 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T01:49:08.436 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T01:49:08.600 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2398.g9c429fb6.el9. 31/36 2024-03-25T01:49:08.630 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/36 2024-03-25T01:49:08.656 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 32/36 2024-03-25T01:49:08.656 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:08.656 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T01:49:08.656 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T01:49:08.656 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T01:49:08.657 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 36/36 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/36 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/36 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 3/36 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 4/36 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/36 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2398.g9c429fb6.el9. 6/36 2024-03-25T01:49:09.892 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T01:49:09.893 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T01:49:09.894 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T01:49:09.895 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T01:49:09.895 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T01:49:09.896 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T01:49:09.896 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T01:49:09.896 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T01:49:09.896 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T01:49:10.105 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 33/36 2024-03-25T01:49:10.131 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 33/36 2024-03-25T01:49:10.131 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:10.131 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T01:49:10.131 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T01:49:10.132 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T01:49:10.132 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:10.573 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 34/36 2024-03-25T01:49:10.596 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 34/36 2024-03-25T01:49:10.596 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:10.596 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T01:49:10.596 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T01:49:10.596 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T01:49:10.596 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:10.823 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 35/36 2024-03-25T01:49:10.846 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 35/36 2024-03-25T01:49:10.847 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:10.847 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T01:49:10.847 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T01:49:10.847 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T01:49:10.847 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:10.913 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 36/36 2024-03-25T01:49:11.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T01:49:11.009 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:11.009 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T01:49:11.010 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T01:49:11.011 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T01:49:11.012 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T01:49:11.012 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T01:49:11.012 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T01:49:11.012 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T01:49:11.012 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T01:49:11.012 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T01:49:11.013 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T01:49:11.013 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T01:49:11.013 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T01:49:11.013 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T01:49:11.013 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T01:49:11.013 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T01:49:11.013 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T01:49:11.014 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T01:49:11.014 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T01:49:11.014 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T01:49:11.014 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:11.014 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:11.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-03-25T01:49:11.939 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:53 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:12.037 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:49:12.089 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:12.090 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:49:12.091 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:12.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-03-25T01:49:12.621 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 36/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 3/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 4/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 5/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2398.g9c429fb6.el9. 6/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T01:49:12.622 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T01:49:12.623 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T01:49:12.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T01:49:12.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T01:49:12.624 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T01:49:12.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T01:49:12.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T01:49:12.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T01:49:12.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T01:49:12.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T01:49:12.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T01:49:12.625 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T01:49:12.725 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:54 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:12.874 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:12.875 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:12.875 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 750 k 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 750 k 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 756 k 2024-03-25T01:49:12.876 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:13.315 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-19.0.0-2398.g9c429fb6.el9.noarch.rpm 1.7 MB/s | 750 kB 00:00 2024-03-25T01:49:13.315 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:13.315 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.7 MB/s | 750 kB 00:00 2024-03-25T01:49:13.316 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:13.326 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:13.326 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:13.398 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:13.750 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T01:49:13.750 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:13.750 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:13.750 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:13.750 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:13.750 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:13.750 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T01:49:13.751 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T01:49:13.752 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T01:49:13.753 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T01:49:13.753 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T01:49:13.753 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T01:49:13.753 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T01:49:13.753 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T01:49:13.753 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:13.753 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:13.840 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:13.927 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:14.105 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-03-25T01:49:14.282 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:14.565 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:14.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:14.565 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:14.565 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:14.565 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:14.565 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:14.683 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:55 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:14.779 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:49:14.795 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T01:49:14.830 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:14.831 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:49:14.831 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:14.901 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-03-25T01:49:15.380 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:57 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:15.464 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:56 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:15.535 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 145 k 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 145 k 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 435 k 2024-03-25T01:49:15.536 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:15.611 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 750 k 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 750 k 2024-03-25T01:49:15.612 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 756 k 2024-03-25T01:49:15.613 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:15.762 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-19.0.0-2398.g9c429f 643 kB/s | 145 kB 00:00 2024-03-25T01:49:15.763 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:15.763 INFO:teuthology.orchestra.run.smithi153.stdout:Total 638 kB/s | 145 kB 00:00 2024-03-25T01:49:15.763 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:15.770 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:15.770 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:15.822 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:15.823 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:15.962 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:16.030 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6. 1/1 2024-03-25T01:49:16.042 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-19.0.0-2398.g9c429fb6.el9.noarch.rpm 1.7 MB/s | 750 kB 00:00 2024-03-25T01:49:16.043 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:16.043 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.7 MB/s | 750 kB 00:00 2024-03-25T01:49:16.043 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:16.048 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:16.048 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6. 1/1 2024-03-25T01:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T01:49:16.054 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:16.055 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:16.055 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:16.130 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:16.575 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:16.647 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:17.062 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:17.142 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6. 1/1 2024-03-25T01:49:17.143 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:17.143 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:17.143 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:17.143 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:17.143 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:17.353 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/1 2024-03-25T01:49:17.353 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:17.353 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:17.354 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:17.354 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:17.354 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:17.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-03-25T01:49:17.613 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T01:49:17.977 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:59 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:18.074 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:49:18.126 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:18.127 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:49:18.127 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:18.191 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:59 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:18.192 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T01:49:18.345 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:18.345 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:18.345 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 145 k 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 145 k 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 435 k 2024-03-25T01:49:18.346 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:18.566 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-19.0.0-2398.g9c429f 659 kB/s | 145 kB 00:00 2024-03-25T01:49:18.567 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:18.567 INFO:teuthology.orchestra.run.smithi083.stdout:Total 654 kB/s | 145 kB 00:00 2024-03-25T01:49:18.567 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:18.574 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:18.574 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:18.626 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:18.626 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:18.757 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:00 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:18.760 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:18.809 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6. 1/1 2024-03-25T01:49:18.833 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6. 1/1 2024-03-25T01:49:18.833 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:18.833 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T01:49:18.833 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:18.906 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:18.907 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-25T01:49:18.907 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:49:18.907 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-25T01:49:18.907 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:18.907 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 3.5 M 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 34 k 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 16 k 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T01:49:18.908 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T01:49:18.909 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T01:49:18.909 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T01:49:18.909 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:18.909 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:18.909 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-03-25T01:49:18.909 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-25T01:49:18.909 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:18.910 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 8.6 M 2024-03-25T01:49:18.910 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 103 M 2024-03-25T01:49:18.910 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:19.813 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6. 1/1 2024-03-25T01:49:19.813 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:19.813 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:19.813 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:19.814 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:19.814 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:19.815 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2398.g9c4 85 kB/s | 16 kB 00:00 2024-03-25T01:49:19.840 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2398.g9c 155 kB/s | 34 kB 00:00 2024-03-25T01:49:20.041 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 238 kB 00:00 2024-03-25T01:49:20.063 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-03-25T01:49:20.108 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-25T01:49:20.233 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-25T01:49:20.317 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 233 kB/s | 19 kB 00:00 2024-03-25T01:49:20.383 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-03-25T01:49:20.450 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-25T01:49:20.493 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 737 kB/s | 31 kB 00:00 2024-03-25T01:49:20.576 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T01:49:20.636 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:01 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:20.734 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:49:20.785 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:20.786 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:49:20.786 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:20.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T01:49:20.926 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-25T01:49:21.268 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): ceph-mgr-dashboard-19.0.0-2398.g9c429f 2.1 MB/s | 3.5 MB 00:01 2024-03-25T01:49:21.272 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:21.272 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.6 MB/s | 8.6 MB 00:02 2024-03-25T01:49:21.290 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:21.308 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:21.308 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:21.407 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:02 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:21.505 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:21.505 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:21.555 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:21.556 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-03-25T01:49:21.556 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:21.556 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-03-25T01:49:21.556 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:21.556 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 3.5 M 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 34 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 16 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:21.557 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:21.558 INFO:teuthology.orchestra.run.smithi083.stdout:====================================================================================== 2024-03-25T01:49:21.558 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-03-25T01:49:21.558 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:21.558 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 8.6 M 2024-03-25T01:49:21.558 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 103 M 2024-03-25T01:49:21.558 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:22.030 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:22.118 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T01:49:22.162 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T01:49:22.204 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2398.g9c4 82 kB/s | 16 kB 00:00 2024-03-25T01:49:22.209 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T01:49:22.256 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T01:49:22.304 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2398.g9c 113 kB/s | 34 kB 00:00 2024-03-25T01:49:22.363 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T01:49:22.372 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T01:49:22.388 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T01:49:22.508 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T01:49:22.601 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T01:49:22.644 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T01:49:22.647 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-25T01:49:22.672 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T01:49:22.705 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2398.g9c429fb6.el9 10/12 2024-03-25T01:49:22.914 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): ceph-mgr-dashboard-19.0.0-2398.g9c429fb 3.9 MB/s | 3.5 MB 00:00 2024-03-25T01:49:23.655 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 19 kB/s | 19 kB 00:01 2024-03-25T01:49:23.763 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2398.g9c429fb6.el 11/12 2024-03-25T01:49:23.805 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noa 12/12 2024-03-25T01:49:25.131 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 200 kB/s | 548 kB 00:02 2024-03-25T01:49:25.285 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noa 12/12 2024-03-25T01:49:25.286 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2398.g9c429fb6.el 1/12 2024-03-25T01:49:25.286 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noa 2/12 2024-03-25T01:49:25.286 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2398.g9c429fb6.el9 3/12 2024-03-25T01:49:25.286 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T01:49:25.286 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T01:49:25.287 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T01:49:25.287 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T01:49:25.287 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T01:49:25.287 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T01:49:25.287 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T01:49:25.287 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T01:49:25.339 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 86 kB/s | 144 kB 00:01 2024-03-25T01:49:25.641 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T01:49:25.641 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:25.641 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:25.641 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T01:49:25.641 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:25.642 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:25.642 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:25.642 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T01:49:25.642 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T01:49:25.642 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T01:49:25.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T01:49:25.642 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T01:49:25.643 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T01:49:25.643 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T01:49:25.643 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T01:49:25.643 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:25.643 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:25.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T01:49:26.331 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 26 kB/s | 31 kB 00:01 2024-03-25T01:49:26.485 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:08 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:26.636 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:26.637 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-25T01:49:26.637 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:49:26.637 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-25T01:49:26.637 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:26.637 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 7.4 M 2024-03-25T01:49:26.637 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:26.638 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:26.639 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================================ 2024-03-25T01:49:26.639 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-25T01:49:26.639 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:26.639 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-03-25T01:49:26.639 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-03-25T01:49:26.639 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:26.984 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 217 kB/s | 32 kB 00:00 2024-03-25T01:49:27.009 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 724 kB/s | 17 kB 00:00 2024-03-25T01:49:27.043 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T01:49:27.218 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T01:49:27.360 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T01:49:27.427 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T01:49:27.452 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-25T01:49:27.594 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 803 kB 00:00 2024-03-25T01:49:27.794 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T01:49:27.845 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 789 kB/s | 195 kB 00:00 2024-03-25T01:49:28.270 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-2 5.2 MB/s | 7.4 MB 00:01 2024-03-25T01:49:28.514 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 59 kB/s | 188 kB 00:03 2024-03-25T01:49:28.761 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 254 kB/s | 245 kB 00:00 2024-03-25T01:49:28.764 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:28.765 INFO:teuthology.orchestra.run.smithi153.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-25T01:49:28.809 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:28.834 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:28.835 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:29.199 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:29.199 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:29.406 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 321 kB/s | 2.0 MB 00:06 2024-03-25T01:49:29.411 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:29.411 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.1 MB/s | 8.6 MB 00:07 2024-03-25T01:49:29.431 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:29.451 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:29.452 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:29.642 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:29.643 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:29.916 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:29.992 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T01:49:30.024 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T01:49:30.117 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T01:49:30.162 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:30.169 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T01:49:30.240 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T01:49:30.283 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T01:49:30.330 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T01:49:30.351 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T01:49:30.361 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T01:49:30.385 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T01:49:30.454 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T01:49:30.462 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T01:49:30.576 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T01:49:30.663 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T01:49:30.715 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T01:49:30.744 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T01:49:30.776 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2398.g9c429fb6.el9 10/12 2024-03-25T01:49:31.152 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T01:49:31.264 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T01:49:31.820 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2398.g9c429fb6.el 11/12 2024-03-25T01:49:31.852 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noa 12/12 2024-03-25T01:49:32.504 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noa 12/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2398.g9c429fb6.el 1/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noa 2/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2398.g9c429fb6.el9 3/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T01:49:33.276 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T01:49:33.277 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T01:49:33.277 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T01:49:33.542 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T01:49:33.567 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429 12/12 2024-03-25T01:49:33.627 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T01:49:33.627 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:33.627 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:33.627 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T01:49:33.627 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:33.627 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:33.628 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:33.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T01:49:34.441 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:15 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:34.591 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:34.592 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 7.4 M 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T01:49:34.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-03-25T01:49:34.594 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:34.595 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 37 M 2024-03-25T01:49:34.595 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 209 M 2024-03-25T01:49:34.595 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:34.990 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-03-25T01:49:35.016 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 719 kB/s | 17 kB 00:00 2024-03-25T01:49:35.049 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T01:49:35.099 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T01:49:35.258 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-25T01:49:35.325 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-25T01:49:35.350 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T01:49:35.684 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-25T01:49:35.886 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 978 kB/s | 195 kB 00:00 2024-03-25T01:49:36.018 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-03-25T01:49:36.177 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-2 5.6 MB/s | 7.4 MB 00:01 2024-03-25T01:49:36.210 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 937 kB/s | 803 kB 00:00 2024-03-25T01:49:36.212 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:36.212 INFO:teuthology.orchestra.run.smithi083.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-25T01:49:36.253 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:36.276 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:36.276 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:36.535 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429 12/12 2024-03-25T01:49:36.535 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429 1/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T01:49:36.536 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T01:49:36.627 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:36.628 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T01:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:37.004 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:37.005 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:37.327 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:37.400 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-03-25T01:49:37.408 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T01:49:37.439 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T01:49:37.536 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T01:49:37.576 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T01:49:37.753 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T01:49:37.783 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T01:49:37.860 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T01:49:37.976 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:38.127 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 49 k 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T01:49:38.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:49:38.129 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-03-25T01:49:38.130 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:38.130 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-03-25T01:49:38.130 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-25T01:49:38.130 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:38.335 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 18 kB 00:00 2024-03-25T01:49:38.360 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 152 kB/s | 24 kB 00:00 2024-03-25T01:49:38.410 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T01:49:38.435 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T01:49:38.461 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T01:49:38.486 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-25T01:49:38.514 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T01:49:38.586 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): ceph-mgr-rook-19.0.0-2398.g9c429fb6.el9 127 kB/s | 49 kB 00:00 2024-03-25T01:49:38.612 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T01:49:38.670 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2024-03-25T01:49:38.695 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 72 kB/s | 15 kB 00:00 2024-03-25T01:49:38.912 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 623 kB/s | 203 kB 00:00 2024-03-25T01:49:39.046 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 672 kB/s | 90 kB 00:00 2024-03-25T01:49:39.121 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 139 kB/s | 59 kB 00:00 2024-03-25T01:49:39.288 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-25T01:49:39.290 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:39.290 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-25T01:49:39.290 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:39.304 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:39.304 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:39.395 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:39.396 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:39.850 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T01:49:39.933 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:40.094 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T01:49:40.155 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T01:49:40.211 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T01:49:40.248 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T01:49:40.289 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T01:49:40.384 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T01:49:40.484 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T01:49:40.541 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T01:49:40.886 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T01:49:40.917 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429 12/12 2024-03-25T01:49:41.274 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T01:49:41.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T01:49:41.355 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T01:49:41.403 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T01:49:41.433 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 13/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 13/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 1/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T01:49:42.220 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T01:49:42.221 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T01:49:42.221 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T01:49:42.221 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T01:49:42.221 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T01:49:42.221 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T01:49:42.588 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:42.589 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:42.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T01:49:43.442 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:25 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:43.492 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429 12/12 2024-03-25T01:49:43.492 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429 1/12 2024-03-25T01:49:43.492 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T01:49:43.492 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T01:49:43.492 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T01:49:43.492 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T01:49:43.492 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T01:49:43.493 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T01:49:43.493 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T01:49:43.493 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T01:49:43.493 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T01:49:43.493 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T01:49:43.591 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:43.592 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:49:43.592 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:49:43.592 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:49:43.592 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:43.592 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 140 k 2024-03-25T01:49:43.592 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-25T01:49:43.593 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:43.594 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.8 M 2024-03-25T01:49:43.594 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-03-25T01:49:43.594 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:43.946 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T01:49:43.947 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T01:49:43.948 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T01:49:43.948 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T01:49:43.948 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T01:49:43.948 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:43.948 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:43.948 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-25T01:49:44.130 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.9 MB/s | 548 kB 00:00 2024-03-25T01:49:44.174 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2024-03-25T01:49:44.200 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): ceph-mgr-cephadm-19.0.0-2398.g9c429fb6.e 316 kB/s | 140 kB 00:00 2024-03-25T01:49:44.275 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-25T01:49:44.300 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 412 kB/s | 51 kB 00:00 2024-03-25T01:49:44.301 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:44.302 INFO:teuthology.orchestra.run.smithi153.stdout:Total 9.6 MB/s | 6.8 MB 00:00 2024-03-25T01:49:44.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-03-25T01:49:44.314 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:44.325 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:44.413 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:44.414 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:44.716 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:44.880 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:25 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:44.890 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T01:49:44.934 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T01:49:45.029 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:45.030 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 49 k 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T01:49:45.031 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-03-25T01:49:45.032 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:45.033 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.0 M 2024-03-25T01:49:45.033 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-03-25T01:49:45.033 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:45.192 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 162 kB/s | 18 kB 00:00 2024-03-25T01:49:45.243 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 150 kB/s | 24 kB 00:00 2024-03-25T01:49:45.293 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T01:49:45.318 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T01:49:45.352 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T01:49:45.377 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 732 kB/s | 43 kB 00:00 2024-03-25T01:49:45.457 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T01:49:45.469 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): ceph-mgr-rook-19.0.0-2398.g9c429fb6.el9 127 kB/s | 49 kB 00:00 2024-03-25T01:49:45.547 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T01:49:45.553 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 161 kB/s | 32 kB 00:00 2024-03-25T01:49:45.578 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2024-03-25T01:49:45.608 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T01:49:45.637 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-25T01:49:45.638 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 6/6 2024-03-25T01:49:45.687 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-25T01:49:45.712 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 841 kB/s | 203 kB 00:00 2024-03-25T01:49:45.871 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.2 MB/s | 1.0 MB 00:00 2024-03-25T01:49:45.875 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:45.875 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-25T01:49:45.875 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:45.887 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:45.888 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:45.979 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:45.979 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:46.433 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 6/6 2024-03-25T01:49:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/6 2024-03-25T01:49:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T01:49:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T01:49:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T01:49:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T01:49:46.494 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:46.657 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T01:49:46.713 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T01:49:46.736 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T01:49:46.736 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:46.737 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:46.768 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T01:49:46.798 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T01:49:46.839 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T01:49:46.932 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T01:49:47.024 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-03-25T01:49:47.033 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T01:49:47.083 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T01:49:47.593 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:29 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:47.748 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:47.748 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:47.748 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:49:47.748 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:47.748 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:47.748 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 890 k 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:47.749 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 969 k 2024-03-25T01:49:47.750 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.8 M 2024-03-25T01:49:47.750 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:47.800 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T01:49:47.855 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T01:49:47.889 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T01:49:47.892 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 551 kB/s | 79 kB 00:00 2024-03-25T01:49:47.953 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T01:49:47.975 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 13/13 2024-03-25T01:49:48.343 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-19.0.0-2398.g9c429fb6.el9.x86_ 1.5 MB/s | 890 kB 00:00 2024-03-25T01:49:48.344 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:48.344 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.6 MB/s | 969 kB 00:00 2024-03-25T01:49:48.345 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:48.365 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:48.365 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:48.438 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:48.438 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:48.625 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:48.720 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 13/13 2024-03-25T01:49:48.720 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 1/13 2024-03-25T01:49:48.720 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T01:49:48.720 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T01:49:48.720 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T01:49:48.720 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T01:49:48.721 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T01:49:48.721 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T01:49:48.721 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T01:49:48.721 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T01:49:48.721 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T01:49:48.721 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T01:49:48.721 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T01:49:48.770 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T01:49:48.834 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/2 2024-03-25T01:49:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T01:49:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:49.086 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T01:49:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T01:49:49.086 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T01:49:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:49.088 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:49.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T01:49:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/2 2024-03-25T01:49:49.634 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/2 2024-03-25T01:49:49.903 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:30 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:49.945 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T01:49:49.945 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:49.945 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:49.946 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T01:49:49.946 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:49.946 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:50.052 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:50.053 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:49:50.053 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:50.053 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:49:50.053 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 140 k 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-03-25T01:49:50.054 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:50.055 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 6.8 M 2024-03-25T01:49:50.055 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 33 M 2024-03-25T01:49:50.055 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:50.260 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-volume 2024-03-25T01:49:50.548 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T01:49:50.573 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 865 kB/s | 248 kB 00:00 2024-03-25T01:49:50.598 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2398.g9c429fb6.e 449 kB/s | 140 kB 00:00 2024-03-25T01:49:50.774 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 288 kB/s | 58 kB 00:00 2024-03-25T01:49:50.799 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 255 kB/s | 51 kB 00:00 2024-03-25T01:49:50.835 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:32 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:50.916 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T01:49:50.916 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:50.917 INFO:teuthology.orchestra.run.smithi083.stdout:Total 7.9 MB/s | 6.8 MB 00:00 2024-03-25T01:49:50.929 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:50.940 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:50.940 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:50.985 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 264 k 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-25T01:49:50.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:50.988 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 332 k 2024-03-25T01:49:50.988 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.5 M 2024-03-25T01:49:50.988 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:51.029 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:51.029 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:51.189 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 342 kB/s | 69 kB 00:00 2024-03-25T01:49:51.214 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-volume-19.0.0-2398.g9c429fb6.el9.no 1.1 MB/s | 264 kB 00:00 2024-03-25T01:49:51.215 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:51.216 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.4 MB/s | 332 kB 00:00 2024-03-25T01:49:51.216 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:51.224 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:51.225 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:51.266 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:51.266 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:51.316 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:51.417 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:51.487 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T01:49:51.533 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T01:49:51.597 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T01:49:51.634 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 2/2 2024-03-25T01:49:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 2/2 2024-03-25T01:49:51.650 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:51.650 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T01:49:51.650 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:52.040 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T01:49:52.120 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T01:49:52.182 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T01:49:52.205 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 6/6 2024-03-25T01:49:52.455 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 1/2 2024-03-25T01:49:52.798 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T01:49:52.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:52.798 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:52.798 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:52.798 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T01:49:52.798 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:52.798 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:52.954 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 6/6 2024-03-25T01:49:52.954 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 1/6 2024-03-25T01:49:52.954 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T01:49:52.954 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T01:49:52.954 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T01:49:52.954 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T01:49:53.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-03-25T01:49:53.249 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:53.250 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:53.532 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-03-25T01:49:53.768 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:53.919 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 127 k 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:53.920 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:49:53.921 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:53.921 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 127 k 2024-03-25T01:49:53.921 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 456 k 2024-03-25T01:49:53.921 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:54.095 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:35 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:54.245 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:54.246 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:54.246 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 890 k 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 969 k 2024-03-25T01:49:54.247 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.8 M 2024-03-25T01:49:54.248 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:54.295 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-19.0.0-2398.g9c429fb6.el9.x86_64 338 kB/s | 127 kB 00:00 2024-03-25T01:49:54.296 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:54.296 INFO:teuthology.orchestra.run.smithi153.stdout:Total 337 kB/s | 127 kB 00:00 2024-03-25T01:49:54.296 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:54.302 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:54.302 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:54.363 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:54.363 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:54.386 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 79 kB 00:00 2024-03-25T01:49:54.509 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:54.578 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-fuse-19.0.0-2398.g9c429fb6.el9.x86_ 2.6 MB/s | 890 kB 00:00 2024-03-25T01:49:54.579 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:54.580 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.8 MB/s | 969 kB 00:00 2024-03-25T01:49:54.580 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:54.600 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:54.600 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:54.601 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:49:54.674 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:54.674 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:54.859 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:54.988 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T01:49:55.048 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:49:55.054 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/2 2024-03-25T01:49:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:49:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:55.362 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:55.362 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:55.363 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:55.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-03-25T01:49:55.689 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/2 2024-03-25T01:49:55.689 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/2 2024-03-25T01:49:55.992 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T01:49:55.992 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:55.992 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:55.992 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T01:49:55.992 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:55.992 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:56.178 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:38 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:56.248 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2024-03-25T01:49:56.281 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:49:56.334 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:56.335 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:49:56.335 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:56.416 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-03-25T01:49:56.822 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:37 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:56.979 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:56.979 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:56.979 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-25T01:49:56.979 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:19.0.0-2398.g9c429fb6.el9 ceph-noarch 264 k 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-25T01:49:56.980 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:56.981 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:38 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:56.981 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 332 k 2024-03-25T01:49:56.981 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.5 M 2024-03-25T01:49:56.981 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:57.108 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 541 kB/s | 69 kB 00:00 2024-03-25T01:49:57.130 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:57.131 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:57.131 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:49:57.131 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:57.131 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:49:57.131 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 32 k 2024-03-25T01:49:57.131 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:57.131 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:49:57.132 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:49:57.132 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:49:57.132 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:57.132 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 32 k 2024-03-25T01:49:57.132 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 142 k 2024-03-25T01:49:57.132 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:49:57.258 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-volume-19.0.0-2398.g9c429fb6.el9.no 951 kB/s | 264 kB 00:00 2024-03-25T01:49:57.259 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:57.260 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-25T01:49:57.260 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:49:57.268 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:49:57.268 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:49:57.305 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:49:57.305 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:49:57.306 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-19.0.0-2398.g9c429fb6.el9.x86_6 182 kB/s | 32 kB 00:00 2024-03-25T01:49:57.307 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:49:57.308 INFO:teuthology.orchestra.run.smithi153.stdout:Total 180 kB/s | 32 kB 00:00 2024-03-25T01:49:57.308 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:49:57.310 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:49:57.311 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:49:57.340 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:49:57.341 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:49:57.475 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:49:57.479 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:49:57.579 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:49:57.647 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T01:49:57.677 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 2/2 2024-03-25T01:49:57.693 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 2/2 2024-03-25T01:49:57.693 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:49:57.693 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T01:49:57.694 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:58.008 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:49:58.270 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:49:58.271 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:58.271 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:49:58.271 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:49:58.271 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:49:58.271 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:58.454 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-03-25T01:49:58.496 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 1/2 2024-03-25T01:49:58.816 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T01:49:58.816 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:58.816 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:49:58.816 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:19.0.0-2398.g9c429fb6.el9.noarch 2024-03-25T01:49:58.816 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T01:49:58.816 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:58.817 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:49:59.017 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:59.114 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:49:59.166 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:59.167 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:49:59.167 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:59.183 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-03-25T01:49:59.220 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-03-25T01:49:59.758 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:40 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:49:59.783 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:49:59.880 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:49:59.906 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:49:59.906 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:59.906 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:49:59.906 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:59.907 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:49:59.907 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 127 k 2024-03-25T01:49:59.907 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:59.907 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:49:59.907 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:49:59.907 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:49:59.907 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:49:59.908 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 127 k 2024-03-25T01:49:59.908 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 456 k 2024-03-25T01:49:59.908 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:49:59.932 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:49:59.933 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:49:59.933 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:49:59.986 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-03-25T01:50:00.131 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-19.0.0-2398.g9c429fb6.el9.x86_64 567 kB/s | 127 kB 00:00 2024-03-25T01:50:00.132 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:00.132 INFO:teuthology.orchestra.run.smithi083.stdout:Total 563 kB/s | 127 kB 00:00 2024-03-25T01:50:00.132 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:50:00.138 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:50:00.138 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:50:00.197 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:50:00.198 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:50:00.350 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:50:00.496 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:00.556 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:42 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:00.654 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:00.705 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:00.706 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:50:00.706 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:00.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-03-25T01:50:00.933 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:01.216 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:01.216 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:01.216 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:50:01.216 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:01.216 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:01.216 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:01.369 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:43 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:01.467 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-03-25T01:50:01.472 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:01.525 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:01.526 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:50:01.526 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:01.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-03-25T01:50:02.042 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:43 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:02.143 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:02.198 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:02.199 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:50:02.199 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:02.241 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:02.265 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-03-25T01:50:02.292 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:02.293 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:50:02.293 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:02.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-03-25T01:50:02.822 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:43 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:02.918 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:02.971 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:02.971 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 32 k 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 32 k 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 142 k 2024-03-25T01:50:02.972 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:50:03.017 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:03.068 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:03.069 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:50:03.069 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:03.128 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-03-25T01:50:03.244 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-19.0.0-2398.g9c429fb6.el9.x86_6 117 kB/s | 32 kB 00:00 2024-03-25T01:50:03.245 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:03.245 INFO:teuthology.orchestra.run.smithi083.stdout:Total 116 kB/s | 32 kB 00:00 2024-03-25T01:50:03.245 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:50:03.248 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:50:03.248 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:50:03.277 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:50:03.277 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:50:03.397 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:50:03.486 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:03.708 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:03.867 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:03.868 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:03.868 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 85 k 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 85 k 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 231 k 2024-03-25T01:50:03.869 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:50:03.879 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:04.182 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:04.182 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:04.182 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:50:04.182 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:04.182 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:04.182 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:04.335 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-03-25T01:50:04.426 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-19.0.0-2398.g9c429fb6.el9.x86_64.rpm 153 kB/s | 85 kB 00:00 2024-03-25T01:50:04.426 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:04.427 INFO:teuthology.orchestra.run.smithi153.stdout:Total 153 kB/s | 85 kB 00:00 2024-03-25T01:50:04.427 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:50:04.434 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:50:04.434 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:50:04.491 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:50:04.492 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:50:04.635 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:50:04.732 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:04.915 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:05.014 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:05.067 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:05.068 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:50:05.068 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:05.123 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-03-25T01:50:05.180 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:05.492 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:05.492 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:05.492 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:50:05.492 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:05.492 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:05.493 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:05.684 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:46 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:05.719 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-03-25T01:50:05.781 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:05.832 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:05.834 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:50:05.834 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:05.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-03-25T01:50:06.297 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:06.446 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:47 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:06.452 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 3.1 M 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:06.453 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.1 M 2024-03-25T01:50:06.454 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 13 M 2024-03-25T01:50:06.454 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:50:06.545 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:06.596 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:06.597 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:50:06.597 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:06.645 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-03-25T01:50:06.837 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-19.0.0-2398.g9c429fb6.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-03-25T01:50:06.837 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:06.838 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-03-25T01:50:06.838 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:50:06.847 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:50:06.847 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:50:06.921 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:50:06.921 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:50:07.210 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:07.308 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:07.346 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:50:07.358 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:07.359 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:50:07.360 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:07.404 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:07.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-03-25T01:50:07.429 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:07.430 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:50:07.430 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T01:50:07.430 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T01:50:07.430 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T01:50:07.430 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:07.972 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:48 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:08.068 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:08.119 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:08.120 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:50:08.120 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:08.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-03-25T01:50:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:08.633 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:50:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:08.633 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:08.634 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:08.737 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:49 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:08.834 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:50:08.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-03-25T01:50:08.885 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:08.886 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:50:08.886 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:08.952 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-03-25T01:50:09.434 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:51 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:09.514 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:50 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:09.583 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 170 k 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:50:09.584 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:09.585 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:50:09.585 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:09.585 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 170 k 2024-03-25T01:50:09.585 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 494 k 2024-03-25T01:50:09.585 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:50:09.662 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:09.663 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:09.663 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-25T01:50:09.663 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:09.663 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:50:09.663 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 85 k 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 85 k 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 231 k 2024-03-25T01:50:09.664 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:50:10.023 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-19.0.0-2398.g9c429fb6.el9.x86_64.rpm 238 kB/s | 85 kB 00:00 2024-03-25T01:50:10.023 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:10.024 INFO:teuthology.orchestra.run.smithi083.stdout:Total 237 kB/s | 85 kB 00:00 2024-03-25T01:50:10.024 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:50:10.030 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:50:10.031 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:50:10.085 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:50:10.085 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:50:10.216 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-19.0.0-2398.g9c429fb6.el9.x86_64.rpm 271 kB/s | 170 kB 00:00 2024-03-25T01:50:10.216 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:10.217 INFO:teuthology.orchestra.run.smithi153.stdout:Total 270 kB/s | 170 kB 00:00 2024-03-25T01:50:10.217 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:50:10.225 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:50:10.225 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:50:10.225 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:50:10.281 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:50:10.281 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:50:10.329 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:50:10.538 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:10.706 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:10.987 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:11.014 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:11.015 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:11.015 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:50:11.015 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:11.015 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:11.015 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:11.225 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-03-25T01:50:11.274 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:11.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:11.274 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:50:11.274 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:11.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:11.274 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:11.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base-debuginfo 2024-03-25T01:50:11.797 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:52 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:11.953 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 3.1 M 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:50:11.954 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:11.955 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.1 M 2024-03-25T01:50:11.955 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 13 M 2024-03-25T01:50:11.955 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:50:12.095 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:12.250 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:12.250 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:12.250 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 69 M 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 23 M 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-25T01:50:12.251 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:12.252 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 92 M 2024-03-25T01:50:12.252 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 502 M 2024-03-25T01:50:12.252 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:50:13.584 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-19.0.0-2398.g9c429fb6.el9.x86_64.rpm 1.9 MB/s | 3.1 MB 00:01 2024-03-25T01:50:13.585 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:13.585 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.9 MB/s | 3.1 MB 00:01 2024-03-25T01:50:13.585 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:50:13.596 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:50:13.596 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:50:13.673 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:50:13.674 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:50:14.103 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:50:14.152 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:14.175 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:14.176 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T01:50:14.176 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T01:50:14.176 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T01:50:14.176 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T01:50:14.176 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:15.273 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:15.273 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:15.274 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:50:15.274 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:15.274 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:15.274 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:15.507 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-03-25T01:50:16.086 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:16.210 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): ceph-debuginfo-19.0.0-2398.g9c429fb6.el9 5.8 MB/s | 23 MB 00:03 2024-03-25T01:50:16.235 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:16.235 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 170 k 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 170 k 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 494 k 2024-03-25T01:50:16.236 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:50:16.738 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-19.0.0-2398.g9c429fb6.el9.x86_64.rpm 340 kB/s | 170 kB 00:00 2024-03-25T01:50:16.739 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:16.739 INFO:teuthology.orchestra.run.smithi083.stdout:Total 339 kB/s | 170 kB 00:00 2024-03-25T01:50:16.739 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:50:16.747 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:50:16.748 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:50:16.803 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:50:16.804 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:50:16.957 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:50:17.053 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:17.458 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:17.780 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:50:17.780 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:17.780 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:50:17.780 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:17.780 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:17.780 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:17.998 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base-debuginfo 2024-03-25T01:50:18.568 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:18.627 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-base-debuginfo-19.0.0-2398.g9c429fb 11 MB/s | 69 MB 00:06 2024-03-25T01:50:18.629 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:18.629 INFO:teuthology.orchestra.run.smithi153.stdout:Total 14 MB/s | 92 MB 00:06 2024-03-25T01:50:18.629 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:50:18.634 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:50:18.634 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:50:18.720 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:18.721 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:18.721 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:50:18.721 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:18.721 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:50:18.721 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 69 M 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 23 M 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 92 M 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 502 M 2024-03-25T01:50:18.722 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:50:19.236 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:50:19.236 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:50:22.436 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:50:22.492 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): ceph-base-debuginfo-19.0.0-2398.g9c429fb 18 MB/s | 69 MB 00:03 2024-03-25T01:50:23.609 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): ceph-debuginfo-19.0.0-2398.g9c429fb6.el9 4.7 MB/s | 23 MB 00:04 2024-03-25T01:50:23.611 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:50:23.611 INFO:teuthology.orchestra.run.smithi083.stdout:Total 19 MB/s | 92 MB 00:04 2024-03-25T01:50:23.611 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:50:23.616 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:50:23.616 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:50:24.219 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:50:24.219 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:50:27.390 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:50:29.642 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/2 2024-03-25T01:50:29.719 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 2/2 2024-03-25T01:50:33.064 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 2/2 2024-03-25T01:50:33.065 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/2 2024-03-25T01:50:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/2 2024-03-25T01:50:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:33.336 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:50:33.336 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:33.337 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:33.337 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:50:33.542 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-common-debuginfo 2024-03-25T01:50:34.107 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:02:16 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:50:34.254 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 443 M 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 443 M 2024-03-25T01:50:34.255 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.8 G 2024-03-25T01:50:34.256 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:50:34.558 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/2 2024-03-25T01:50:34.626 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 2/2 2024-03-25T01:50:37.985 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 2/2 2024-03-25T01:50:37.986 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/2 2024-03-25T01:50:38.261 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2/2 2024-03-25T01:50:38.261 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:38.261 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:50:38.261 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:38.261 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:50:38.261 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:38.261 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:50:38.470 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-common-debuginfo 2024-03-25T01:50:39.049 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:20 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:50:39.210 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 443 M 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:50:39.211 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:50:39.212 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 443 M 2024-03-25T01:50:39.212 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.8 G 2024-03-25T01:50:39.212 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:51:00.488 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-common-debuginfo-19.0.0-2398.g9c429fb6.el9 17 MB/s | 443 MB 00:26 2024-03-25T01:51:00.489 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:51:00.489 INFO:teuthology.orchestra.run.smithi153.stdout:Total 17 MB/s | 443 MB 00:26 2024-03-25T01:51:00.489 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:51:00.493 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:51:00.493 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:51:03.125 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:51:03.125 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:51:39.543 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-common-debuginfo-19.0.0-2398.g9c429fb6.el9 7.4 MB/s | 443 MB 01:00 2024-03-25T01:51:39.544 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:51:39.544 INFO:teuthology.orchestra.run.smithi083.stdout:Total 7.4 MB/s | 443 MB 01:00 2024-03-25T01:51:39.544 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:51:39.548 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:51:39.548 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:51:42.180 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:51:42.180 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:51:48.144 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:51:48.507 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:51:55.499 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:51:55.756 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:51:55.756 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:55.757 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:51:55.757 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:51:55.757 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:55.757 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:51:56.005 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T01:51:56.597 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:38 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:51:56.750 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.5 M 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:56.751 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.5 M 2024-03-25T01:51:56.752 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 6.6 M 2024-03-25T01:51:56.752 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:51:57.259 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-23 2.9 MB/s | 1.5 MB 00:00 2024-03-25T01:51:57.260 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:51:57.260 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.9 MB/s | 1.5 MB 00:00 2024-03-25T01:51:57.260 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:51:57.264 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:51:57.264 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:51:57.306 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:51:57.306 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:51:57.564 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:51:57.657 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2398. 1/1 2024-03-25T01:51:58.067 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-2398. 1/1 2024-03-25T01:51:58.353 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2398. 1/1 2024-03-25T01:51:58.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:58.354 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:51:58.354 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:51:58.354 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:58.354 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:51:58.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T01:51:59.101 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:03:41 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:51:59.252 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 178 M 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:51:59.253 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:51:59.254 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:51:59.254 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:51:59.254 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 178 M 2024-03-25T01:51:59.254 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 764 M 2024-03-25T01:51:59.254 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:52:07.253 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-radosgw-debuginfo-19.0.0-2398.g9c429fb6.el 22 MB/s | 178 MB 00:07 2024-03-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:52:07.254 INFO:teuthology.orchestra.run.smithi153.stdout:Total 22 MB/s | 178 MB 00:08 2024-03-25T01:52:07.255 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:52:07.259 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:52:08.341 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:52:08.342 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:52:26.743 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:52:26.841 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x 1/1 2024-03-25T01:52:26.978 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:52:27.070 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:52:31.491 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x 1/1 2024-03-25T01:52:31.760 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x 1/1 2024-03-25T01:52:31.761 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:52:31.761 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:52:31.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:52:31.761 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:52:31.761 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:52:31.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test-debuginfo 2024-03-25T01:52:32.542 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:52:32.691 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 680 M 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:52:32.692 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:52:32.693 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:52:32.693 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:52:32.693 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 680 M 2024-03-25T01:52:32.693 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 3.1 G 2024-03-25T01:52:32.693 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:52:35.557 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:52:35.812 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:52:35.812 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:35.812 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:52:35.812 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:52:35.812 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:35.812 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:52:36.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T01:52:36.611 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:52:36.766 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:52:36.767 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:52:36.767 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:52:36.767 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:52:36.767 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:52:36.767 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.5 M 2024-03-25T01:52:36.767 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:36.767 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:52:36.768 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================== 2024-03-25T01:52:36.768 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:52:36.768 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:36.768 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.5 M 2024-03-25T01:52:36.768 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 6.6 M 2024-03-25T01:52:36.768 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:52:37.517 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-23 2.0 MB/s | 1.5 MB 00:00 2024-03-25T01:52:37.517 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:52:37.518 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.9 MB/s | 1.5 MB 00:00 2024-03-25T01:52:37.518 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:52:37.524 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:52:37.524 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:52:37.584 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:52:37.585 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:52:37.860 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:52:37.948 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2398. 1/1 2024-03-25T01:52:38.347 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-2398. 1/1 2024-03-25T01:52:38.628 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2398. 1/1 2024-03-25T01:52:38.629 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:38.629 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:52:38.629 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:52:38.629 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:38.629 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:52:38.818 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T01:52:39.385 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:52:39.534 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:52:39.534 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 178 M 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 178 M 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 764 M 2024-03-25T01:52:39.535 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:52:53.602 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-radosgw-debuginfo-19.0.0-2398.g9c429fb6.el 13 MB/s | 178 MB 00:14 2024-03-25T01:52:53.603 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:52:53.603 INFO:teuthology.orchestra.run.smithi083.stdout:Total 13 MB/s | 178 MB 00:14 2024-03-25T01:52:53.603 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:52:53.608 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:52:53.608 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:52:54.685 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:52:54.685 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:53:08.292 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-test-debuginfo-19.0.0-2398.g9c429fb6.el9.x 19 MB/s | 680 MB 00:35 2024-03-25T01:53:08.292 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:53:08.293 INFO:teuthology.orchestra.run.smithi153.stdout:Total 19 MB/s | 680 MB 00:35 2024-03-25T01:53:08.293 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:53:08.298 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:53:08.298 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:53:12.306 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:53:12.306 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:53:13.248 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:53:13.440 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x 1/1 2024-03-25T01:53:18.234 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x 1/1 2024-03-25T01:53:18.488 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x 1/1 2024-03-25T01:53:18.488 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:53:18.488 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:53:18.488 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:53:18.488 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:53:18.488 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:53:18.687 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test-debuginfo 2024-03-25T01:53:19.250 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:05:00 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:53:19.402 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:53:19.403 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:53:19.403 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:53:19.403 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:53:19.403 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:53:19.403 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 680 M 2024-03-25T01:53:19.403 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:53:19.403 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:53:19.404 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:53:19.404 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:53:19.404 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:53:19.404 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 680 M 2024-03-25T01:53:19.404 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 3.1 G 2024-03-25T01:53:19.404 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:53:48.876 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-test-debuginfo-19.0.0-2398.g9c429fb6.el9.x 23 MB/s | 680 MB 00:29 2024-03-25T01:53:48.876 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:53:48.877 INFO:teuthology.orchestra.run.smithi083.stdout:Total 23 MB/s | 680 MB 00:29 2024-03-25T01:53:48.877 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:53:48.882 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:53:48.883 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:53:52.887 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:53:52.888 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:54:24.185 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:54:24.273 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:54:34.063 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:54:34.330 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:54:34.330 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:34.330 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:54:34.330 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:54:34.330 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:34.330 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:54:34.612 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base-debuginfo 2024-03-25T01:54:35.174 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:54:35.273 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:54:35.325 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:54:35.326 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-25T01:54:35.326 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:54:35.395 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T01:54:35.959 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:54:36.110 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:54:36.110 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 20 M 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 20 M 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 104 M 2024-03-25T01:54:36.111 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:54:37.235 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-mgr-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 18 MB/s | 20 MB 00:01 2024-03-25T01:54:37.236 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:54:37.236 INFO:teuthology.orchestra.run.smithi153.stdout:Total 18 MB/s | 20 MB 00:01 2024-03-25T01:54:37.237 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:54:37.240 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:54:37.241 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:54:37.392 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:54:37.393 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:54:39.667 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:54:39.784 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:54:40.754 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:54:41.013 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:54:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:41.014 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:54:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:54:41.014 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:41.014 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:54:41.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T01:54:41.769 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:23 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:54:41.918 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:54:41.919 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:41.919 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:54:41.919 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:41.919 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 54 M 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 54 M 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 195 M 2024-03-25T01:54:41.920 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:54:44.437 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-mds-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 21 MB/s | 54 MB 00:02 2024-03-25T01:54:44.438 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:54:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:Total 21 MB/s | 54 MB 00:02 2024-03-25T01:54:44.439 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:54:44.444 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:54:44.444 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:54:44.790 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:54:44.790 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:54:50.105 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:54:50.194 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:54:51.732 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:54:51.995 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:54:51.995 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:51.995 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:54:51.995 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:54:51.995 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:51.995 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:54:52.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T01:54:52.754 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:54:52.903 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:54:52.903 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 99 M 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 99 M 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 451 M 2024-03-25T01:54:52.904 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:54:58.131 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-mon-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 19 MB/s | 99 MB 00:05 2024-03-25T01:54:58.132 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:54:58.132 INFO:teuthology.orchestra.run.smithi153.stdout:Total 19 MB/s | 99 MB 00:05 2024-03-25T01:54:58.133 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:54:58.136 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:54:58.136 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:54:58.752 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:54:58.752 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:55:04.651 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:55:04.752 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:55:09.226 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:55:09.337 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:12.195 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:55:12.451 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:55:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:55:12.451 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:55:12.451 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:55:12.691 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T01:55:13.256 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:06:55 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:55:13.406 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 290 M 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:55:13.407 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:55:13.408 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:55:13.408 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 290 M 2024-03-25T01:55:13.408 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.2 G 2024-03-25T01:55:13.408 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:55:14.087 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:55:14.340 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:55:14.340 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:14.341 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:55:14.341 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:55:14.341 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:14.341 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:55:14.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base-debuginfo 2024-03-25T01:55:15.157 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:56 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:55:15.253 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 is already installed. 2024-03-25T01:55:15.304 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:55:15.305 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-03-25T01:55:15.305 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:55:15.362 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T01:55:15.931 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:06:56 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:55:16.081 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 20 M 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:16.086 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:55:16.087 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:16.087 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 20 M 2024-03-25T01:55:16.087 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 104 M 2024-03-25T01:55:16.087 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:55:20.565 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mgr-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 4.5 MB/s | 20 MB 00:04 2024-03-25T01:55:20.566 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:55:20.566 INFO:teuthology.orchestra.run.smithi083.stdout:Total 4.5 MB/s | 20 MB 00:04 2024-03-25T01:55:20.566 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:55:20.570 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:55:20.570 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:55:20.723 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:55:20.723 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:55:22.999 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:55:23.155 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:24.176 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:24.431 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:24.431 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:24.431 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:55:24.432 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:55:24.432 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:24.432 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:55:24.598 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T01:55:25.158 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:07:06 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:55:25.307 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:55:25.307 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:25.307 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:55:25.307 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:25.307 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 54 M 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 54 M 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 195 M 2024-03-25T01:55:25.308 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:55:26.806 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-osd-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 22 MB/s | 290 MB 00:13 2024-03-25T01:55:26.807 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:55:26.807 INFO:teuthology.orchestra.run.smithi153.stdout:Total 22 MB/s | 290 MB 00:13 2024-03-25T01:55:26.807 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:55:26.812 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:55:26.812 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:55:28.009 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mds-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 20 MB/s | 54 MB 00:02 2024-03-25T01:55:28.009 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:55:28.010 INFO:teuthology.orchestra.run.smithi083.stdout:Total 20 MB/s | 54 MB 00:02 2024-03-25T01:55:28.010 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:55:28.014 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:55:28.014 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:55:28.362 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:55:28.362 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:55:28.550 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:55:28.550 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:55:33.676 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:55:33.806 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:35.309 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:35.570 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:35.570 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:35.570 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:55:35.570 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:55:35.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:35.571 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:55:35.754 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T01:55:36.319 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:07:17 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:55:36.468 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:55:36.468 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:36.468 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:55:36.468 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:36.468 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:55:36.468 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 99 M 2024-03-25T01:55:36.468 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:36.469 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:55:36.469 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:36.469 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:55:36.469 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:36.469 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 99 M 2024-03-25T01:55:36.469 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 451 M 2024-03-25T01:55:36.469 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:55:42.269 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-mon-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 17 MB/s | 99 MB 00:05 2024-03-25T01:55:42.270 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:55:42.270 INFO:teuthology.orchestra.run.smithi083.stdout:Total 17 MB/s | 99 MB 00:05 2024-03-25T01:55:42.271 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:55:42.275 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:55:42.275 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:55:42.891 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:55:42.891 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:55:53.203 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:55:53.316 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:56.429 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:56.692 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:55:56.693 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:56.693 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:55:56.693 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:55:56.693 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:56.693 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:55:56.858 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T01:55:57.420 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:07:38 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:55:57.569 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:55:57.570 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:57.570 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:55:57.570 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:57.570 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:55:57.570 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 290 M 2024-03-25T01:55:57.570 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:57.570 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:55:57.571 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:55:57.571 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:55:57.571 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:55:57.571 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 290 M 2024-03-25T01:55:57.571 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.2 G 2024-03-25T01:55:57.571 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:55:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:55:58.369 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:56:05.787 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:56:06.074 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:56:06.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:06.074 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:06.074 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:06.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:06.074 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:06.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T01:56:06.888 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:07:48 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:07.037 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:07.038 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:07.038 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 6.1 M 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.1 M 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 29 M 2024-03-25T01:56:07.039 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:07.722 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-fuse-debuginfo-19.0.0-2398.g9c429fb6.el9.x 8.9 MB/s | 6.1 MB 00:00 2024-03-25T01:56:07.722 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:07.723 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.9 MB/s | 6.1 MB 00:00 2024-03-25T01:56:07.723 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:07.726 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:07.726 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:07.794 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:07.794 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:08.536 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:08.644 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:56:09.207 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:56:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:56:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:09.484 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:09.484 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:09.484 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:09.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel-debuginfo 2024-03-25T01:56:10.265 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:07:52 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:10.414 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:10.415 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:10.415 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:10.415 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 109 k 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 109 k 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 440 k 2024-03-25T01:56:10.416 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:10.758 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-debuginfo-19.0.0-2398.g9c429fb6. 318 kB/s | 109 kB 00:00 2024-03-25T01:56:10.759 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:10.759 INFO:teuthology.orchestra.run.smithi153.stdout:Total 317 kB/s | 109 kB 00:00 2024-03-25T01:56:10.759 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:10.762 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:10.762 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:10.796 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:10.796 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:10.908 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:11.014 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:56:11.425 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:56:11.689 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:56:11.689 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:11.689 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:11.689 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:11.690 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:11.690 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:11.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T01:56:12.452 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:07:54 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:12.602 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 5.5 M 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 5.5 M 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 25 M 2024-03-25T01:56:12.603 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:13.270 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs2-debuginfo-19.0.0-2398.g9c429fb6.el9. 8.2 MB/s | 5.5 MB 00:00 2024-03-25T01:56:13.270 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:13.271 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.2 MB/s | 5.5 MB 00:00 2024-03-25T01:56:13.271 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:13.274 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:13.274 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:13.339 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:13.340 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:14.015 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:14.118 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:56:14.636 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:56:14.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:56:14.924 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:14.924 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:14.924 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:14.925 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:14.925 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:15.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2-debuginfo 2024-03-25T01:56:15.713 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:07:57 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:15.862 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:15.863 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:15.863 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:15.863 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:15.863 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 45 M 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 45 M 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 257 M 2024-03-25T01:56:15.864 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:18.222 INFO:teuthology.orchestra.run.smithi153.stdout:librados2-debuginfo-19.0.0-2398.g9c429fb6.el9.x 19 MB/s | 45 MB 00:02 2024-03-25T01:56:18.223 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:18.223 INFO:teuthology.orchestra.run.smithi153.stdout:Total 19 MB/s | 45 MB 00:02 2024-03-25T01:56:18.224 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:18.228 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:18.228 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:18.538 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:18.538 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:23.719 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:23.812 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:56:25.720 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:56:25.991 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:56:25.991 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:25.991 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:25.991 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:25.991 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:25.991 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:26.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1-debuginfo 2024-03-25T01:56:26.766 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:08 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:26.913 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:26.913 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:26.913 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:26.913 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:26.913 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 52 M 2024-03-25T01:56:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:26.914 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:26.914 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:26.914 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:26.914 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:26.915 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 52 M 2024-03-25T01:56:26.915 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 309 M 2024-03-25T01:56:26.915 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:29.399 INFO:teuthology.orchestra.run.smithi153.stdout:librbd1-debuginfo-19.0.0-2398.g9c429fb6.el9.x86 21 MB/s | 52 MB 00:02 2024-03-25T01:56:29.400 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:29.401 INFO:teuthology.orchestra.run.smithi153.stdout:Total 21 MB/s | 52 MB 00:02 2024-03-25T01:56:29.401 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:29.405 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:29.405 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:29.755 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:29.756 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:35.821 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:35.923 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:56:38.123 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:56:38.395 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:56:38.396 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:38.396 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:38.396 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:38.396 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:38.396 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:38.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T01:56:39.186 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:21 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:39.338 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 615 k 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:39.339 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:39.340 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:39.340 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:39.340 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 615 k 2024-03-25T01:56:39.340 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.1 M 2024-03-25T01:56:39.340 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:39.618 INFO:teuthology.orchestra.run.smithi153.stdout:python3-cephfs-debuginfo-19.0.0-2398.g9c429fb6. 2.2 MB/s | 615 kB 00:00 2024-03-25T01:56:39.619 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:39.619 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.1 MB/s | 615 kB 00:00 2024-03-25T01:56:39.619 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:39.622 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:39.622 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:39.660 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:39.660 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:39.822 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:39.925 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:56:40.353 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:56:40.628 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:56:40.628 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:40.628 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:40.628 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:40.628 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:40.628 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:40.834 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados-debuginfo 2024-03-25T01:56:41.399 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:23 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:41.548 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:41.548 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 770 k 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 770 k 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.7 M 2024-03-25T01:56:41.549 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:41.988 INFO:teuthology.orchestra.run.smithi153.stdout:python3-rados-debuginfo-19.0.0-2398.g9c429fb6.e 1.7 MB/s | 770 kB 00:00 2024-03-25T01:56:41.988 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:41.989 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.7 MB/s | 770 kB 00:00 2024-03-25T01:56:41.989 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:41.992 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:41.992 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:42.030 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:42.030 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:42.211 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:42.320 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9. 1/1 2024-03-25T01:56:42.703 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9. 1/1 2024-03-25T01:56:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9. 1/1 2024-03-25T01:56:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:42.987 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:42.987 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:43.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T01:56:43.757 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:25 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:43.905 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.1 M 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:43.906 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:43.907 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.1 M 2024-03-25T01:56:43.907 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 4.3 M 2024-03-25T01:56:43.907 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:44.207 INFO:teuthology.orchestra.run.smithi153.stdout:python3-rbd-debuginfo-19.0.0-2398.g9c429fb6.el9 3.7 MB/s | 1.1 MB 00:00 2024-03-25T01:56:44.208 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:44.208 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.7 MB/s | 1.1 MB 00:00 2024-03-25T01:56:44.208 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:44.212 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:44.212 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:44.252 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:44.252 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:44.469 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:44.582 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:56:44.977 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:45.254 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:45.255 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:45.465 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T01:56:46.023 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:28 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:46.171 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 337 k 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:46.172 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:46.173 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 337 k 2024-03-25T01:56:46.173 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 1.0 M 2024-03-25T01:56:46.173 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:46.427 INFO:teuthology.orchestra.run.smithi153.stdout:python3-rgw-debuginfo-19.0.0-2398.g9c429fb6.el9 1.3 MB/s | 337 kB 00:00 2024-03-25T01:56:46.428 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:46.428 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-03-25T01:56:46.428 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:46.434 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:46.434 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:46.468 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:46.468 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:46.605 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:46.718 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:56:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:56:47.367 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:56:47.367 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:47.367 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:47.367 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:47.367 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:47.367 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:47.577 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T01:56:48.135 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:30 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:48.286 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:48.286 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:48.286 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:48.286 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 517 k 2024-03-25T01:56:48.286 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:48.287 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:48.287 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:48.287 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:48.287 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:48.287 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 517 k 2024-03-25T01:56:48.287 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.6 M 2024-03-25T01:56:48.287 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:48.564 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 1.8 MB/s | 517 kB 00:00 2024-03-25T01:56:48.564 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:48.565 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.8 MB/s | 517 kB 00:00 2024-03-25T01:56:48.565 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:48.568 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:48.568 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:48.604 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:48.604 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:56:48.769 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:56:48.880 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:56:49.275 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:56:49.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:56:49.543 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:49.543 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:56:49.543 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:56:49.543 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:49.543 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:56:49.735 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T01:56:50.292 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:32 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:56:50.441 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 52 M 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:56:50.442 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 52 M 2024-03-25T01:56:50.443 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 299 M 2024-03-25T01:56:50.443 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:56:59.150 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-debuginfo-19.0.0-2398.g9c429fb6.el9. 6.0 MB/s | 52 MB 00:08 2024-03-25T01:56:59.150 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:56:59.150 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.0 MB/s | 52 MB 00:08 2024-03-25T01:56:59.151 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:56:59.155 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:56:59.155 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:56:59.494 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:56:59.494 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:57:01.889 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-osd-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 4.5 MB/s | 290 MB 01:04 2024-03-25T01:57:01.890 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:57:01.890 INFO:teuthology.orchestra.run.smithi083.stdout:Total 4.5 MB/s | 290 MB 01:04 2024-03-25T01:57:01.891 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:57:01.895 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:57:01.895 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:57:03.628 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:57:03.628 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:57:05.481 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:57:05.564 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:57:07.592 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:57:07.859 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:57:07.859 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:57:07.859 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:57:07.859 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:57:07.860 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:57:07.860 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:57:08.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T01:57:08.641 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:08:50 ago on Mon 25 Mar 2024 01:48:18 AM UTC. 2024-03-25T01:57:08.791 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.3 M 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 1.3 M 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 5.7 M 2024-03-25T01:57:08.792 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-25T01:57:09.242 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-debuginfo-19.0.0-2398.g9c429fb6.el9.x86 2.9 MB/s | 1.3 MB 00:00 2024-03-25T01:57:09.243 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:57:09.243 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.9 MB/s | 1.3 MB 00:00 2024-03-25T01:57:09.243 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-25T01:57:09.247 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-25T01:57:09.247 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-25T01:57:09.287 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-25T01:57:09.287 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-25T01:57:09.527 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-25T01:57:09.641 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:57:10.061 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:57:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:57:10.342 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:57:10.343 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-25T01:57:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:57:10.343 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:57:10.343 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-25T01:57:10.567 DEBUG:teuthology.parallel:result is None 2024-03-25T01:57:33.425 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:57:33.586 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:57:41.254 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:57:41.533 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:57:41.533 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:41.534 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:57:41.534 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:57:41.534 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:41.534 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:57:41.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T01:57:42.268 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:23 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:57:42.416 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:57:42.416 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:42.416 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:57:42.416 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:42.416 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:57:42.416 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 6.1 M 2024-03-25T01:57:42.416 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:42.417 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:57:42.417 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:42.417 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:57:42.417 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:42.417 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 6.1 M 2024-03-25T01:57:42.417 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 29 M 2024-03-25T01:57:42.417 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:57:42.939 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-fuse-debuginfo-19.0.0-2398.g9c429fb6.el9.x 12 MB/s | 6.1 MB 00:00 2024-03-25T01:57:42.940 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:57:42.940 INFO:teuthology.orchestra.run.smithi083.stdout:Total 12 MB/s | 6.1 MB 00:00 2024-03-25T01:57:42.941 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:57:42.944 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:57:42.944 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:57:43.012 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:57:43.012 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:57:43.755 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:57:43.846 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:57:44.407 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:57:44.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:57:44.661 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:44.661 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:57:44.661 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:57:44.661 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:44.661 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:57:44.849 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel-debuginfo 2024-03-25T01:57:45.409 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:26 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:57:45.554 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:57:45.554 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:45.554 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:57:45.554 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:45.554 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 109 k 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 109 k 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 440 k 2024-03-25T01:57:45.555 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:57:45.876 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-debuginfo-19.0.0-2398.g9c429fb6. 340 kB/s | 109 kB 00:00 2024-03-25T01:57:45.876 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:57:45.877 INFO:teuthology.orchestra.run.smithi083.stdout:Total 338 kB/s | 109 kB 00:00 2024-03-25T01:57:45.877 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:57:45.880 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:57:45.880 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:57:45.914 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:57:45.914 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:57:46.034 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:57:46.133 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:57:46.504 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:57:46.763 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:57:46.763 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:46.763 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:57:46.764 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:57:46.764 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:46.764 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:57:46.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T01:57:47.499 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:28 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:57:47.648 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:57:47.648 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:47.648 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:57:47.648 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 5.5 M 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 5.5 M 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 25 M 2024-03-25T01:57:47.649 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:57:48.179 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs2-debuginfo-19.0.0-2398.g9c429fb6.el9. 10 MB/s | 5.5 MB 00:00 2024-03-25T01:57:48.180 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:57:48.180 INFO:teuthology.orchestra.run.smithi083.stdout:Total 10 MB/s | 5.5 MB 00:00 2024-03-25T01:57:48.180 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:57:48.184 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:57:48.184 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:57:48.248 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:57:48.248 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:57:48.919 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:57:49.011 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:57:49.582 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:57:49.844 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:57:49.844 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:49.844 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:57:49.844 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:57:49.844 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:49.844 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:57:50.026 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2-debuginfo 2024-03-25T01:57:50.585 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:31 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 45 M 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:50.734 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:57:50.735 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:57:50.735 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:57:50.735 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:57:50.735 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 45 M 2024-03-25T01:57:50.735 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 257 M 2024-03-25T01:57:50.735 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:57:53.374 INFO:teuthology.orchestra.run.smithi083.stdout:librados2-debuginfo-19.0.0-2398.g9c429fb6.el9.x 17 MB/s | 45 MB 00:02 2024-03-25T01:57:53.374 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:57:53.375 INFO:teuthology.orchestra.run.smithi083.stdout:Total 17 MB/s | 45 MB 00:02 2024-03-25T01:57:53.375 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:57:53.379 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:57:53.379 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:57:53.688 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:57:53.688 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:57:58.844 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:57:58.948 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:58:00.852 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:58:01.109 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_ 1/1 2024-03-25T01:58:01.109 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:01.109 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:01.109 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:01.109 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:01.109 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:01.276 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1-debuginfo 2024-03-25T01:58:01.845 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:42 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:01.996 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:01.997 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:01.997 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:01.997 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 52 M 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 52 M 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 309 M 2024-03-25T01:58:01.998 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:04.910 INFO:teuthology.orchestra.run.smithi083.stdout:librbd1-debuginfo-19.0.0-2398.g9c429fb6.el9.x86 18 MB/s | 52 MB 00:02 2024-03-25T01:58:04.911 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:04.911 INFO:teuthology.orchestra.run.smithi083.stdout:Total 18 MB/s | 52 MB 00:02 2024-03-25T01:58:04.911 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:04.916 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:04.916 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:05.262 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:05.262 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:11.268 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:11.359 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:58:13.557 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:58:13.824 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:58:13.824 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:13.824 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:13.824 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:13.824 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:13.825 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:14.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T01:58:14.565 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:55 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:14.720 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 615 k 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 615 k 2024-03-25T01:58:14.721 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.1 M 2024-03-25T01:58:14.722 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:15.263 INFO:teuthology.orchestra.run.smithi083.stdout:python3-cephfs-debuginfo-19.0.0-2398.g9c429fb6. 1.1 MB/s | 615 kB 00:00 2024-03-25T01:58:15.263 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:15.264 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.1 MB/s | 615 kB 00:00 2024-03-25T01:58:15.264 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:15.267 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:15.267 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:15.303 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:15.304 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:15.469 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:15.569 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:58:15.989 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:58:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9 1/1 2024-03-25T01:58:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:16.258 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:16.258 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:16.258 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:16.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados-debuginfo 2024-03-25T01:58:16.994 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:09:57 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:17.141 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:17.142 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:17.142 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:17.142 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 770 k 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 770 k 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.7 M 2024-03-25T01:58:17.143 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:17.608 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rados-debuginfo-19.0.0-2398.g9c429fb6.e 1.6 MB/s | 770 kB 00:00 2024-03-25T01:58:17.608 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:17.609 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.6 MB/s | 770 kB 00:00 2024-03-25T01:58:17.609 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:17.612 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:17.612 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:17.649 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:17.650 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:17.827 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:17.923 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9. 1/1 2024-03-25T01:58:18.328 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9. 1/1 2024-03-25T01:58:18.600 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9. 1/1 2024-03-25T01:58:18.600 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:18.600 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:18.600 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:18.601 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:18.601 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:18.808 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T01:58:19.366 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:10:00 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:19.513 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.1 M 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:19.514 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:19.515 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:19.515 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.1 M 2024-03-25T01:58:19.515 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 4.3 M 2024-03-25T01:58:19.515 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:19.820 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rbd-debuginfo-19.0.0-2398.g9c429fb6.el9 3.7 MB/s | 1.1 MB 00:00 2024-03-25T01:58:19.820 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:19.821 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.7 MB/s | 1.1 MB 00:00 2024-03-25T01:58:19.821 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:19.824 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:19.824 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:19.864 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:19.864 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:20.084 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:20.185 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:58:20.576 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:58:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:58:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:20.832 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:20.832 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:20.832 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:21.028 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T01:58:21.589 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:10:02 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:21.736 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:21.737 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:21.737 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:21.737 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:21.737 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 337 k 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 337 k 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.0 M 2024-03-25T01:58:21.738 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:22.231 INFO:teuthology.orchestra.run.smithi083.stdout:python3-rgw-debuginfo-19.0.0-2398.g9c429fb6.el9 684 kB/s | 337 kB 00:00 2024-03-25T01:58:22.232 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:22.232 INFO:teuthology.orchestra.run.smithi083.stdout:Total 682 kB/s | 337 kB 00:00 2024-03-25T01:58:22.232 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:22.235 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:22.236 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:22.271 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:22.271 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:22.410 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:22.514 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:58:22.903 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:58:23.174 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x8 1/1 2024-03-25T01:58:23.174 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:23.174 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:23.175 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:23.175 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:23.175 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:23.373 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T01:58:23.935 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:10:04 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:24.084 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:24.085 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:24.085 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:24.085 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:24.085 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:24.085 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 517 k 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 517 k 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 2.6 M 2024-03-25T01:58:24.086 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:24.701 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-debuginfo-19.0.0-2398.g9c429fb6.el9.x8 840 kB/s | 517 kB 00:00 2024-03-25T01:58:24.702 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:24.702 INFO:teuthology.orchestra.run.smithi083.stdout:Total 838 kB/s | 517 kB 00:00 2024-03-25T01:58:24.702 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:24.705 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:24.706 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:24.742 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:24.743 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:24.903 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:25.000 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:58:25.400 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:58:25.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_6 1/1 2024-03-25T01:58:25.664 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:25.664 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:25.664 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:25.664 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:25.664 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:25.848 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T01:58:26.409 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:10:07 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:26.559 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 52 M 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:26.560 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:26.561 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 52 M 2024-03-25T01:58:26.561 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 299 M 2024-03-25T01:58:26.561 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:29.529 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-debuginfo-19.0.0-2398.g9c429fb6.el9. 17 MB/s | 52 MB 00:02 2024-03-25T01:58:29.530 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:29.530 INFO:teuthology.orchestra.run.smithi083.stdout:Total 17 MB/s | 52 MB 00:02 2024-03-25T01:58:29.530 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:29.534 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:29.535 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:29.872 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:29.872 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:35.810 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:35.913 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:58:37.986 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:58:38.253 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86 1/1 2024-03-25T01:58:38.253 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:38.253 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:38.254 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:38.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:38.254 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:38.441 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T01:58:39.002 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:10:20 ago on Mon 25 Mar 2024 01:48:19 AM UTC. 2024-03-25T01:58:39.151 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2398.g9c429fb6.el9 ceph 1.3 M 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-25T01:58:39.152 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-03-25T01:58:39.153 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-03-25T01:58:39.153 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:39.153 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.3 M 2024-03-25T01:58:39.153 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 5.7 M 2024-03-25T01:58:39.153 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-25T01:58:39.640 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-debuginfo-19.0.0-2398.g9c429fb6.el9.x86 2.6 MB/s | 1.3 MB 00:00 2024-03-25T01:58:39.641 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-25T01:58:39.641 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.6 MB/s | 1.3 MB 00:00 2024-03-25T01:58:39.641 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-25T01:58:39.644 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-25T01:58:39.644 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-25T01:58:39.685 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-25T01:58:39.686 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-25T01:58:39.928 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-25T01:58:40.031 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:58:40.482 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:58:40.737 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 1/1 2024-03-25T01:58:40.737 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:40.737 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-25T01:58:40.737 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-debuginfo-2:19.0.0-2398.g9c429fb6.el9.x86_64 2024-03-25T01:58:40.737 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:40.738 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-25T01:58:40.923 DEBUG:teuthology.parallel:result is None 2024-03-25T01:58:40.923 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-25T01:58:41.152 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T01:58:41.172 INFO:teuthology.orchestra.run.smithi083.stdout:19.0.0-2398.g9c429fb6.el9 2024-03-25T01:58:41.173 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2398.g9c429fb6.el9 2024-03-25T01:58:41.173 INFO:teuthology.task.install:The correct ceph version 19.0.0-2398.g9c429fb6 is installed. 2024-03-25T01:58:41.175 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-25T01:58:41.313 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T01:58:41.338 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2398.g9c429fb6.el9 2024-03-25T01:58:41.338 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2398.g9c429fb6.el9 2024-03-25T01:58:41.339 INFO:teuthology.task.install:The correct ceph version 19.0.0-2398.g9c429fb6 is installed. 2024-03-25T01:58:41.340 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T01:58:41.341 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:41.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T01:58:41.368 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:58:41.368 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T01:58:41.410 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T01:58:41.411 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:41.411 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T01:58:41.438 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T01:58:41.504 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:58:41.504 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T01:58:41.533 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T01:58:41.603 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T01:58:41.603 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:41.603 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T01:58:41.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T01:58:41.697 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:58:41.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T01:58:41.725 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T01:58:41.794 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T01:58:41.794 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:41.794 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T01:58:41.821 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T01:58:41.887 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:58:41.887 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T01:58:41.915 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T01:58:41.982 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T01:58:42.065 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T01:58:42.066 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-03-25T01:58:42.068 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-03-25T01:58:42.094 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T01:58:42.095 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T01:58:42.135 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T01:58:42.161 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T01:58:42.161 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T01:58:42.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T01:58:42.239 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T01:58:42.239 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-25T01:58:42.239 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770', 'branch': 'wip-yuri10-testing-2024-03-24-1159', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7620770', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2024-03-24-1159'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2024-03-24-1159', 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 108, 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri10-testing-2024-03-24-1159', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f', 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObskKB0hMJpfUxtnW8i01ugVMaSIKTiGMPE9pwaWD74qdcrRNDF80UoV9ZtlOqvva7+TeHF3DSRg//B/c0wnVw=', 'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCSw63IetpNrST60r4TDQMhJcu7ayY4c4BA+U3CukOW/GwnEviD1/7HU+6Ttzt5Qh8lFWze4ACmLVo6Q1ugJG2k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-24_22:19:24', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T01:58:42.240 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T01:58:42.268 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T01:58:42.283 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T01:58:42.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T01:58:42.351 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:42.351 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T01:58:42.401 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T01:58:42.402 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 01:50:14.421854606 +0000 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 01:47:26.145279070 +0000 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 01:47:26.145279070 +0000 2024-03-25T01:58:42.457 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 01:47:26.145279070 +0000 2024-03-25T01:58:42.457 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T01:58:42.521 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T01:58:42.522 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T01:58:42.522 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000218293 s, 2.3 MB/s 2024-03-25T01:58:42.523 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T01:58:42.579 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-03-25T01:58:42.633 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 01:50:14.421854606 +0000 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 01:47:26.491271207 +0000 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 01:47:26.491271207 +0000 2024-03-25T01:58:42.634 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 01:47:26.491271207 +0000 2024-03-25T01:58:42.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T01:58:42.698 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T01:58:42.698 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T01:58:42.698 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000233883 s, 2.2 MB/s 2024-03-25T01:58:42.700 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T01:58:42.755 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 01:50:14.422854582 +0000 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 01:47:26.825263617 +0000 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 01:47:26.825263617 +0000 2024-03-25T01:58:42.810 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 01:47:26.825263617 +0000 2024-03-25T01:58:42.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T01:58:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T01:58:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T01:58:42.875 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000206192 s, 2.5 MB/s 2024-03-25T01:58:42.876 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T01:58:42.932 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-03-25T01:58:42.985 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-25 01:50:14.422854582 +0000 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-25 01:47:27.177255618 +0000 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-25 01:47:27.177255618 +0000 2024-03-25T01:58:42.986 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-25 01:47:27.177255618 +0000 2024-03-25T01:58:42.986 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T01:58:43.051 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-25T01:58:43.051 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-25T01:58:43.051 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000204485 s, 2.5 MB/s 2024-03-25T01:58:43.052 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T01:58:43.108 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T01:58:43.108 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:58:43.108 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T01:58:43.126 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T01:58:43.126 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 01:50:07.683776306 +0000 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 01:47:28.240414257 +0000 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 01:47:28.240414257 +0000 2024-03-25T01:58:43.183 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 01:47:28.240414257 +0000 2024-03-25T01:58:43.183 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T01:58:43.251 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T01:58:43.251 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T01:58:43.251 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000275484 s, 1.9 MB/s 2024-03-25T01:58:43.252 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T01:58:43.310 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 01:50:07.683776306 +0000 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 01:47:28.582407746 +0000 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 01:47:28.582407746 +0000 2024-03-25T01:58:43.366 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 01:47:28.582407746 +0000 2024-03-25T01:58:43.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T01:58:43.433 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T01:58:43.433 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T01:58:43.433 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000225089 s, 2.3 MB/s 2024-03-25T01:58:43.434 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T01:58:43.490 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-25T01:58:43.546 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T01:58:43.546 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:43.546 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T01:58:43.546 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:43.546 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:43.547 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 01:50:07.684776288 +0000 2024-03-25T01:58:43.547 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 01:47:28.933401064 +0000 2024-03-25T01:58:43.547 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 01:47:28.933401064 +0000 2024-03-25T01:58:43.547 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 01:47:28.933401064 +0000 2024-03-25T01:58:43.547 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T01:58:43.613 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T01:58:43.613 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T01:58:43.613 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000170873 s, 3.0 MB/s 2024-03-25T01:58:43.614 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T01:58:43.669 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-25 01:50:07.684776288 +0000 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-25 01:47:29.264394763 +0000 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-25 01:47:29.264394763 +0000 2024-03-25T01:58:43.725 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-25 01:47:29.264394763 +0000 2024-03-25T01:58:43.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T01:58:43.790 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-25T01:58:43.791 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-25T01:58:43.791 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000225592 s, 2.3 MB/s 2024-03-25T01:58:43.791 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T01:58:43.847 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T01:58:43.847 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T01:58:43.847 INFO:tasks.ceph:Generating config... 2024-03-25T01:58:43.848 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T01:58:43.848 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T01:58:43.848 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T01:58:43.848 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T01:58:43.849 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T01:58:43.849 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-25T01:58:43.849 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T01:58:43.849 INFO:tasks.ceph:[global] ms type = async 2024-03-25T01:58:43.849 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T01:58:43.849 INFO:tasks.ceph:[global] osd op thread timeout = 150 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T01:58:43.849 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T01:58:43.850 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T01:58:43.850 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T01:58:43.850 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T01:58:43.850 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T01:58:43.850 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-03-25T01:58:43.851 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T01:58:43.852 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T01:58:43.852 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T01:58:43.852 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T01:58:43.852 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T01:58:43.852 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T01:58:43.909 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T01:58:43.912 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T01:58:43.956 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T01:58:43.983 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.83'), ('mon.c', '[v2:172.21.15.83:3301,v1:172.21.15.83:6790]'), ('mon.b', '172.21.15.153')] 2024-03-25T01:58:43.983 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83,[v2:172.21.15.83:3301,v1:172.21.15.83:6790],172.21.15.153', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T01:58:43.984 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:43.984 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T01:58:44.039 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --addv c '[v2:172.21.15.83:3301,v1:172.21.15.83:6790]' --add b 172.21.15.153 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:44.124 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:44.124 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:58:44.124 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = quincy 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 17 (quincy) 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:58:44.125 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T01:58:44.126 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T01:58:44.141 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 86338f78-de3e-4243-afbe-49cfeed24a78... 2024-03-25T01:58:44.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T01:58:44.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T01:58:44.231 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = cephx 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.232 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.233 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.234 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = "172.21.15.83,[v2:172.21.15.83:3301,v1:172.21.15.83:6790],172.21.15.153" 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: mon client directed command retry = 5 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: mon election default strategy = 1 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: ms inject socket failures = 5000 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: ms type = async 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: osd heartbeat grace = 80 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: osd op thread timeout = 150 2024-03-25T01:58:44.235 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-03-25T01:58:44.236 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-03-25T01:58:44.237 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: bdev async discard = True 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: bdev enable discard = True 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore block size = 96636764160 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore fsck on mount = True 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: bluestore zero block detection = True 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 1 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 1 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T01:58:44.238 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 20 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 10 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd full ratio = 0.9 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd nearfull ratio = 0.8 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd class default list = * 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd class load list = * 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd client watch timeout = 120 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify cached snaps = True 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify missing on start = True 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd failsafe full ratio = 0.95 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd fast shutdown = False 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock override recovery settings = True 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd memory target = 1610612736 2024-03-25T01:58:44.239 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = bluestore 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue = debug_random 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: osd op queue cut off = debug_random 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: osd shutdown pgref assert = True 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.240 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 20 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T01:58:44.241 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon min osdmap epochs = 50 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd crush smoke test = False 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune interval = 2 2024-03-25T01:58:44.242 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune min = 15 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: mon osdmap full prune txsize = 2 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: mon scrub interval = 300 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on pool no app = False 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: paxos service trim min = 10 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T01:58:44.243 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi083.stdout: debug objecter = 20 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi083.stdout: debug rados = 20 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.c] 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.b] 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-03-25T01:58:44.244 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.245 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-03-25T01:58:44.246 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-03-25T01:58:44.247 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "172.21.15.83,[v2:172.21.15.83:3301,v1:172.21.15.83:6790],172.21.15.153" 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 1 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 5000 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: osd heartbeat grace = 80 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: osd op thread timeout = 150 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-03-25T01:58:44.248 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.249 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: bdev async discard = True 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: bdev enable discard = True 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-03-25T01:58:44.250 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 1 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 1 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: osd class default list = * 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: osd class load list = * 2024-03-25T01:58:44.251 INFO:teuthology.orchestra.run.smithi153.stdout: osd client watch timeout = 120 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd fast shutdown = False 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd memory target = 1610612736 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: osd shutdown pgref assert = True 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.252 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-03-25T01:58:44.253 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T01:58:44.254 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 50 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd crush smoke test = False 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on pool no app = False 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 10 2024-03-25T01:58:44.255 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: debug objecter = 20 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout: debug rados = 20 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-03-25T01:58:44.256 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-03-25T01:58:44.258 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T01:58:44.258 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T01:58:44.354 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T01:58:44.354 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:44.354 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T01:58:44.369 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:44.369 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T01:58:44.423 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T01:58:44.423 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:44.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T01:58:44.423 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T01:58:44.502 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:58:44.502 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:44.557 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T01:58:44.557 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:58:44.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T01:58:44.557 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T01:58:44.597 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:58:44.597 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:44.651 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T01:58:44.651 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T01:58:44.652 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T01:58:44.706 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T01:58:44.709 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T01:58:44.800 INFO:teuthology.orchestra.run.smithi153.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T01:58:44.803 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T01:58:44.803 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T01:58:44.803 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T01:58:44.845 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T01:58:44.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T01:58:44.903 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T01:58:44.918 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T01:58:44.918 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T01:58:44.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T01:58:44.945 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T01:58:44.945 INFO:tasks.ceph:role: osd.0 2024-03-25T01:58:44.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T01:58:44.945 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:58:45.020 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:58:45.532 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T01:58:45.534 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T01:58:45.535 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T01:58:45.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T01:58:45.689 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T01:58:45.757 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T01:58:45.757 INFO:tasks.ceph:role: osd.1 2024-03-25T01:58:45.757 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T01:58:45.757 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T01:58:45.831 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:58:45.832 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:58:46.289 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T01:58:46.291 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T01:58:46.291 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T01:58:46.385 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T01:58:46.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T01:58:46.482 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T01:58:46.482 INFO:tasks.ceph:role: osd.2 2024-03-25T01:58:46.482 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T01:58:46.482 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T01:58:46.556 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:58:46.556 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:58:46.556 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:58:46.556 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:58:46.556 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:58:46.557 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:58:46.557 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:58:46.557 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:58:46.557 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:58:46.557 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:58:47.042 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T01:58:47.044 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T01:58:47.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T01:58:47.191 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T01:58:47.223 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T01:58:47.289 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T01:58:47.289 INFO:tasks.ceph:role: osd.3 2024-03-25T01:58:47.289 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T01:58:47.290 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:58:47.364 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:58:47.365 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:58:47.365 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:58:47.886 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-03-25T01:58:47.888 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-03-25T01:58:47.888 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T01:58:48.023 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T01:58:48.059 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:48.158+0000 7eff902a0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T01:58:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:48.158+0000 7eff902a0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T01:58:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:48.158+0000 7eff902a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T01:58:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:48.158+0000 7eff902a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T01:58:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:48.158+0000 7eff902a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T01:58:48.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:48.159+0000 7eff902a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T01:58:51.012 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T01:58:51.044 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:51.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:51.133+0000 7f55f5ad9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T01:58:51.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:51.133+0000 7f55f5ad9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T01:58:51.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:51.133+0000 7f55f5ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T01:58:51.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:51.133+0000 7f55f5ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T01:58:51.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:51.133+0000 7f55f5ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T01:58:51.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:51.134+0000 7f55f5ad9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T01:58:53.980 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T01:58:54.012 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:54.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:54.100+0000 7faec2666640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T01:58:54.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:54.100+0000 7faec2666640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T01:58:54.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:54.100+0000 7faec2666640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T01:58:54.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:54.100+0000 7faec2666640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T01:58:54.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:54.100+0000 7faec2666640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T01:58:54.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:54.101+0000 7faec2666640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T01:58:56.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T01:58:56.984 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:58:57.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:57.073+0000 7fe0c52bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T01:58:57.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:57.073+0000 7fe0c52bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T01:58:57.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:57.073+0000 7fe0c52bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T01:58:57.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:57.074+0000 7fe0c52bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T01:58:57.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:57.074+0000 7fe0c52bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T01:58:57.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:58:57.074+0000 7fe0c52bd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T01:58:59.915 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T01:58:59.946 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T01:58:59.979 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T01:58:59.979 INFO:tasks.ceph:role: osd.4 2024-03-25T01:58:59.979 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T01:58:59.979 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T01:59:00.057 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:59:00.057 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:59:00.057 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:59:00.058 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:59:00.058 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:59:00.058 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:59:00.058 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:59:00.058 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:59:00.058 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:59:00.058 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:59:00.569 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T01:59:00.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T01:59:00.572 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T01:59:00.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T01:59:00.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T01:59:00.828 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T01:59:00.828 INFO:tasks.ceph:role: osd.5 2024-03-25T01:59:00.828 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T01:59:00.828 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T01:59:00.904 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:59:00.904 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:59:00.904 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:59:00.904 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:59:00.905 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:59:00.905 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:59:00.905 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:59:00.905 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:59:00.905 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:59:00.905 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:59:01.390 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T01:59:01.392 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T01:59:01.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T01:59:01.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T01:59:01.526 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T01:59:01.592 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T01:59:01.592 INFO:tasks.ceph:role: osd.6 2024-03-25T01:59:01.592 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T01:59:01.593 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T01:59:01.667 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:59:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:59:01.667 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:59:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:59:01.668 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:59:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:59:01.668 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:59:01.668 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:59:01.668 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:59:01.668 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:59:02.182 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T01:59:02.184 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T01:59:02.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T01:59:02.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T01:59:02.335 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T01:59:02.403 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T01:59:02.403 INFO:tasks.ceph:role: osd.7 2024-03-25T01:59:02.403 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T01:59:02.403 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T01:59:02.477 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T01:59:02.478 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T01:59:02.987 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-03-25T01:59:02.989 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-03-25T01:59:02.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T01:59:03.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T01:59:03.153 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:59:03.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:03.252+0000 7f5338cd9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T01:59:03.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:03.252+0000 7f5338cd9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T01:59:03.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:03.252+0000 7f5338cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T01:59:03.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:03.252+0000 7f5338cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T01:59:03.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:03.252+0000 7f5338cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T01:59:03.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:03.253+0000 7f5338cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T01:59:06.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T01:59:06.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:59:06.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:06.230+0000 7fd85ec01640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T01:59:06.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:06.230+0000 7fd85ec01640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T01:59:06.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:06.230+0000 7fd85ec01640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T01:59:06.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:06.230+0000 7fd85ec01640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T01:59:06.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:06.230+0000 7fd85ec01640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T01:59:06.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:06.230+0000 7fd85ec01640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T01:59:09.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T01:59:09.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:59:09.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:09.199+0000 7fdcc9c10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T01:59:09.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:09.199+0000 7fdcc9c10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T01:59:09.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:09.199+0000 7fdcc9c10640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T01:59:09.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:09.199+0000 7fdcc9c10640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T01:59:09.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:09.199+0000 7fdcc9c10640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T01:59:09.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:09.200+0000 7fdcc9c10640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T01:59:12.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T01:59:12.075 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:59:12.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:12.162+0000 7f526f624640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T01:59:12.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:12.162+0000 7f526f624640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T01:59:12.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:12.162+0000 7f526f624640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T01:59:12.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:12.162+0000 7f526f624640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T01:59:12.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:12.162+0000 7f526f624640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T01:59:12.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:12.163+0000 7f526f624640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T01:59:15.004 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T01:59:15.035 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T01:59:15.036 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:15.036 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T01:59:15.068 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:15.068 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T01:59:15.140 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:15.140 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T01:59:15.210 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:15.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T01:59:15.280 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:15.280 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T01:59:15.349 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:15.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T01:59:15.380 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:15.381 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T01:59:15.452 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:15.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T01:59:15.523 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:15.523 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T01:59:15.593 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:15.593 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T01:59:15.662 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:15.663 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T01:59:15.681 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:15.681 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T01:59:15.719 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T01:59:15.719 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T01:59:15.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T01:59:15.762 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.y] 2024-03-25T01:59:15.762 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBU2gBmuiIbKhAAoWpYN6qXZ7Rt11EDJ1IpwA== 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBY2gBm4+KCCRAAv5mfcvX8exIpwjX+bw+FjA== 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBb2gBm0PkHCBAA5qdPNZYUOky/TvVK1zB7Ww== 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBe2gBmwNsUBhAA4nql0pz7iIrq2OiJs4XwAQ== 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.3] 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBh2gBmYvt5BBAA4PXUSGBAphRfSDrpb0htag== 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBU2gBmpMy1LxAAiUH8zKuQ5ViGM8F/V9BhxQ== 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.4] 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBn2gBmnegqDxAAPw6dpY18f8zaKvDeBbCZjw== 2024-03-25T01:59:15.763 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.5] 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBq2gBmWrLDDRAAydMkYh3vWRsCqlP0oREcCA== 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.6] 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBt2gBmBv/yCxAAr15RTiRYQeFupgr82Iqueg== 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.7] 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBw2gBmS0q6CRAACBq/lfsN9ROBepjDka6cnw== 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBU2gBml3tjMhAAYuiTb41gEiSIsrwuRz4loQ== 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout:[client.1] 2024-03-25T01:59:15.764 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQBU2gBmDGfQNRAADx6dvMi1xlpwF4Sra/MDwg== 2024-03-25T01:59:15.785 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.y] 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBU2gBmuiIbKhAAoWpYN6qXZ7Rt11EDJ1IpwA== 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBY2gBm4+KCCRAAv5mfcvX8exIpwjX+bw+FjA== 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBb2gBm0PkHCBAA5qdPNZYUOky/TvVK1zB7Ww== 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBe2gBmwNsUBhAA4nql0pz7iIrq2OiJs4XwAQ== 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBh2gBmYvt5BBAA4PXUSGBAphRfSDrpb0htag== 2024-03-25T01:59:15.786 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBU2gBmpMy1LxAAiUH8zKuQ5ViGM8F/V9BhxQ== 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBn2gBmnegqDxAAPw6dpY18f8zaKvDeBbCZjw== 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBq2gBmWrLDDRAAydMkYh3vWRsCqlP0oREcCA== 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.6] 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBt2gBmBv/yCxAAr15RTiRYQeFupgr82Iqueg== 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.7] 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBw2gBmS0q6CRAACBq/lfsN9ROBepjDka6cnw== 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBU2gBml3tjMhAAYuiTb41gEiSIsrwuRz4loQ== 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-03-25T01:59:15.787 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBU2gBmDGfQNRAADx6dvMi1xlpwF4Sra/MDwg== 2024-03-25T01:59:15.788 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T01:59:15.808 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T01:59:15.883 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:15.885 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:15.938 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:15.940 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:15.992 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:15.994 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.047 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.050 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T01:59:16.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T01:59:16.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.158 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.210 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.321 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.323 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T01:59:16.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T01:59:16.376 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T01:59:16.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T01:59:16.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T01:59:16.488 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T01:59:16.488 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T01:59:16.515 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T01:59:16.612 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T01:59:16.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T01:59:16.705 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T01:59:16.801 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T01:59:16.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T01:59:16.855 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T01:59:16.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T01:59:16.979 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:59:16.982 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T01:59:17.034 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T01:59:17.035 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T01:59:17.035 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T01:59:17.035 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T01:59:17.037 INFO:tasks.ceph.mon.a:Started 2024-03-25T01:59:17.037 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T01:59:17.037 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T01:59:17.037 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T01:59:17.039 INFO:tasks.ceph.mon.c:Started 2024-03-25T01:59:17.039 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T01:59:17.040 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T01:59:17.040 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T01:59:17.077 INFO:tasks.ceph.mon.b:Started 2024-03-25T01:59:17.077 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T01:59:17.077 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T01:59:17.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T01:59:17.080 INFO:tasks.ceph.mgr.y:Started 2024-03-25T01:59:17.080 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T01:59:17.080 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T01:59:17.082 INFO:tasks.ceph.mgr.x:Started 2024-03-25T01:59:17.082 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T01:59:17.082 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T01:59:17.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.191+0000 7f0e8797e640 10 monclient: get_monmap_and_config 2024-03-25T01:59:17.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.191+0000 7f0e8797e640 10 monclient: build_initial_monmap 2024-03-25T01:59:17.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.191+0000 7f0e8797e640 10 monclient: monmap: 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:59:17.192381+0000 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:59:17.192381+0000 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:17.193 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:17.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.192+0000 7f0e8797e640 1 Processor -- start 2024-03-25T01:59:17.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.192+0000 7f0e8797e640 1 -- start start 2024-03-25T01:59:17.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.192+0000 7f0e8797e640 10 monclient: init 2024-03-25T01:59:17.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:17.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 0x7f0e80146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient(hunting): picked mon.noname-a con 0x7f0e800a9b20 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e801403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e800b3330 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e800a9f00 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e800abb00 con 0x7f0e800a9f00 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800abde0 con 0x7f0e800a9b20 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800b2e80 con 0x7f0e800b3330 2024-03-25T01:59:17.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:17.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8797e640 10 monclient(hunting): authenticate will time out at 1276.382356s 2024-03-25T01:59:17.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8571a640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 msgr2=0x7f0e80146ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-25T01:59:17.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e85f1b640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7f0e800a9f00 legacy=0x7f0e80143140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T01:59:17.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e8571a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 0x7f0e80146ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:17.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e84f19640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e801403b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:17.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.193+0000 7f0e84f19640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e801403b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:17.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.394+0000 7f0e8571a640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 msgr2=0x7f0e80146ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-25T01:59:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.394+0000 7f0e85f1b640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7f0e800a9f00 legacy=0x7f0e80143140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T01:59:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.394+0000 7f0e8571a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 0x7f0e80146ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-25T01:59:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.394+0000 7f0e84f19640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e801403b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:17.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.394+0000 7f0e84f19640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e801403b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-25T01:59:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.795+0000 7f0e8571a640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 msgr2=0x7f0e80146ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-25T01:59:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.795+0000 7f0e85f1b640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7f0e800a9f00 legacy=0x7f0e80143140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T01:59:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.795+0000 7f0e8571a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 0x7f0e80146ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-25T01:59:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.795+0000 7f0e84f19640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e801403b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:17.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:17.795+0000 7f0e84f19640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e801403b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-25T01:59:18.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:18.596+0000 7f0e85f1b640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7f0e800a9f00 legacy=0x7f0e80143140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.83:6789/0 2024-03-25T01:59:18.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:18.596+0000 7f0e8571a640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 msgr2=0x7f0e80146ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3300/0 2024-03-25T01:59:18.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:18.596+0000 7f0e8571a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 0x7f0e80146ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-25T01:59:18.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:18.596+0000 7f0e84f19640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e801403b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:18.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:18.596+0000 7f0e84f19640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e801403b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-25T01:59:20.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient(hunting): tick 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 monclient(hunting): continuing hunt 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e801403b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e801403b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 -- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 msgr2=0x7f0e80146ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e800a9b20 0x7f0e80146ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 -- >> v1:172.21.15.83:6789/0 conn(0x7f0e800a9f00 legacy=0x7f0e80143140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient: _add_conns ranks=[2,1] 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 0x7f0e64005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0e64003690 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e640061c0 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e800aaef0 con 0x7f0e640061c0 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:20.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f0e800abde0 con 0x7f0e64003690 2024-03-25T01:59:20.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e76ffd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:20.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e8571a640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:20.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e8571a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:20.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.193+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:20.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.394+0000 7f0e8571a640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:20.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.394+0000 7f0e8571a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-25T01:59:20.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.394+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:20.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.795+0000 7f0e8571a640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:20.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.795+0000 7f0e8571a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-25T01:59:20.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:20.795+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:21.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:21.596+0000 7f0e8571a640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:21.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:21.596+0000 7f0e8571a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-25T01:59:21.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:21.596+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:23.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): tick 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 monclient(hunting): continuing hunt 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64003690 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e85f1b640 1 -- reap_dead start 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e64009f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0e800b3330 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e640061c0 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:23.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 0x7f0e64005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6400a480 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e800abb00 con 0x7f0e640061c0 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800abde0 con 0x7f0e800b3330 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f0e800b2e80 con 0x7f0e6400a480 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e76ffd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e85f1b640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e8571a640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e64009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e8571a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e64009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:23.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.193+0000 7f0e85f1b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:23.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.394+0000 7f0e85f1b640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:23.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.394+0000 7f0e85f1b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-25T01:59:23.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.394+0000 7f0e8571a640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e64009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:23.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.394+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:23.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.394+0000 7f0e8571a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e64009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-25T01:59:23.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.795+0000 7f0e85f1b640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:23.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.795+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:23.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.795+0000 7f0e85f1b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-25T01:59:23.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.795+0000 7f0e8571a640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e64009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:23.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:23.795+0000 7f0e8571a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e64009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-25T01:59:24.014 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:00:06.807 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T01:59:24.142 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:00:06.978 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T01:59:24.142 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:00:06.978 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T01:59:24.320 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:00:07.113 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T01:59:24.461 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:00:07.297 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T01:59:24.461 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:00:07.297 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T01:59:24.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:24.596+0000 7f0e85f1b640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.83:3301/0 2024-03-25T01:59:24.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:24.596+0000 7f0e85f1b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-25T01:59:24.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:24.596+0000 7f0e8571a640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e64009f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-03-25T01:59:24.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:24.596+0000 7f0e84f19640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-03-25T01:59:24.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:24.596+0000 7f0e8571a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e64009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-25T01:59:26.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient(hunting): tick 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 monclient(hunting): continuing hunt 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 -- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e6400a480 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 msgr2=0x7f0e64009f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0e800b3330 0x7f0e64009f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f0e640061c0 legacy=0x7f0e64008680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient: _add_conns ranks=[3,0] 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6400ab20 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e6400f740 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:26.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e800aaef0 con 0x7f0e6400f740 2024-03-25T01:59:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800abde0 con 0x7f0e6400ab20 2024-03-25T01:59:26.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.193+0000 7f0e76ffd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:26.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.653+0000 7f0e8571a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.653+0000 7f0e84f19640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f0e6400f740 0x7f0e64011b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:34508/0 (socket says 172.21.15.83:34508) 2024-03-25T01:59:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.653+0000 7f0e84f19640 1 -- 172.21.15.83:0/1336518635 learned_addr learned my addr 172.21.15.83:0/1336518635 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T01:59:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.654+0000 7f0e8571a640 10 monclient(hunting): get_auth_request con 0x7f0e6400ab20 auth_method 0 2024-03-25T01:59:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.654+0000 7f0e8571a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.654+0000 7f0e8571a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:26.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.654+0000 7f0e8571a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.678+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T01:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.678+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T01:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.678+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:26.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.678+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.908+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.908+0000 7f0e8571a640 10 monclient(hunting): get_auth_request con 0x7f0e6400ab20 auth_method 0 2024-03-25T01:59:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.908+0000 7f0e8571a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:26.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.908+0000 7f0e8571a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:26.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.908+0000 7f0e8571a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T01:59:26.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.910+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T01:59:26.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.910+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T01:59:26.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.910+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:26.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:26.910+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-25T01:59:27.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.316+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:27.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.317+0000 7f0e8571a640 10 monclient(hunting): get_auth_request con 0x7f0e6400ab20 auth_method 0 2024-03-25T01:59:27.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.317+0000 7f0e8571a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:27.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.317+0000 7f0e8571a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:27.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.317+0000 7f0e8571a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T01:59:27.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.318+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T01:59:27.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.318+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T01:59:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.318+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:27.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:27.318+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-25T01:59:28.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.126+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:28.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.128+0000 7f0e8571a640 10 monclient(hunting): get_auth_request con 0x7f0e6400ab20 auth_method 0 2024-03-25T01:59:28.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.128+0000 7f0e8571a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:28.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.128+0000 7f0e8571a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:28.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.128+0000 7f0e8571a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T01:59:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.130+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T01:59:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.130+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T01:59:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.130+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:28.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:28.130+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-25T01:59:29.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): tick 2024-03-25T01:59:29.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T01:59:29.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 monclient(hunting): continuing hunt 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 msgr2=0x7f0e6400f130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.83:3300/0 conn(0x7f0e6400ab20 0x7f0e6400f130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 -- 172.21.15.83:0/1336518635 >> v1:172.21.15.83:6789/0 conn(0x7f0e6400f740 legacy=0x7f0e64011b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e85f1b640 1 -- 172.21.15.83:0/1336518635 reap_dead start 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 0x7f0e64005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f0e64009f20 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e64013450 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.153:3300/0 conn(0x7f0e64013830 0x7f0e64013ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0e64013830 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 -- 172.21.15.83:0/1336518635 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e800abb00 con 0x7f0e64013450 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:29.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 -- 172.21.15.83:0/1336518635 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f0e800abde0 con 0x7f0e64013830 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 1 -- 172.21.15.83:0/1336518635 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f0e800b2e80 con 0x7f0e64009f20 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e76ffd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 0x7f0e64005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.153:3300/0 conn(0x7f0e64013830 0x7f0e64013ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e8571a640 10 monclient(hunting): get_auth_request con 0x7f0e64009f20 auth_method 0 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e8571a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e8571a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.194+0000 7f0e8571a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.195+0000 7f0e85f1b640 10 monclient(hunting): get_auth_request con 0x7f0e64013830 auth_method 0 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.195+0000 7f0e85f1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.195+0000 7f0e85f1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:29.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.195+0000 7f0e85f1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:29.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.196+0000 7f0e85f1b640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.153:3300/0 conn(0x7f0e64013830 msgr2=0x7f0e64013ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T01:59:29.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.196+0000 7f0e85f1b640 1 -- 172.21.15.83:0/1336518635 >> v2:172.21.15.153:3300/0 conn(0x7f0e64013830 msgr2=0x7f0e64013ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T01:59:29.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.196+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.153:3300/0 conn(0x7f0e64013830 0x7f0e64013ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:29.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.196+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.153:3300/0 conn(0x7f0e64013830 0x7f0e64013ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:29.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.200+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T01:59:29.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.200+0000 7f0e8571a640 1 -- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T01:59:29.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.200+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 0x7f0e64005ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:29.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.200+0000 7f0e8571a640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 0x7f0e64005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:29.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.312+0000 7f0e77fff640 1 -- 172.21.15.83:0/1336518635 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905806420 0 0) 0x7f0e800abb00 con 0x7f0e64013450 2024-03-25T01:59:29.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.312+0000 7f0e77fff640 10 monclient(hunting): _negotiate 2024-03-25T01:59:29.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.312+0000 7f0e77fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:29.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.312+0000 7f0e77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:29.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.312+0000 7f0e77fff640 10 monclient(hunting): my global_id is 4099 2024-03-25T01:59:29.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.312+0000 7f0e77fff640 1 -- 172.21.15.83:0/1336518635 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e6c003520 con 0x7f0e64013450 2024-03-25T01:59:29.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.418+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/1336518635 >> v2:172.21.15.153:3300/0 conn(0x7f0e64013830 0x7f0e64013ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:29.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.419+0000 7f0e85f1b640 10 monclient(hunting): get_auth_request con 0x7f0e64013830 auth_method 0 2024-03-25T01:59:29.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.419+0000 7f0e85f1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:29.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.419+0000 7f0e85f1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:29.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.419+0000 7f0e85f1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T01:59:29.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.426+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:29.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.426+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:29.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.426+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 1 -- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 msgr2=0x7f0e64005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 0x7f0e64005ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 1 -- 172.21.15.83:0/1336518635 >> v1:172.21.15.153:6789/0 conn(0x7f0e64013450 legacy=0x7f0e64011b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 10 monclient: _finish_hunting 0 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 1 monclient: found mon.noname-c 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:29.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T01:59:29.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.445+0000 7f0e85f1b640 1 -- 172.21.15.83:0/1336518635 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e640096c0 con 0x7f0e64013830 2024-03-25T01:59:29.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.468+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e64013ca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e80146b40 tx=0x7f0e7802c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=323aa725087cf684 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 1 -- 172.21.15.83:0/1336518635 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78039070 con 0x7f0e64013830 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: dump: 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:29.511 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: _finish_auth 0 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: _check_auth_tickets 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 1 -- 172.21.15.83:0/1336518635 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e78031710 con 0x7f0e64013830 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 5 monclient: authenticate success, global_id 4105 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 4 set_mon_vals no callback set 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 1 -- 172.21.15.83:0/1336518635 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78031b00 con 0x7f0e64013830 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:29.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e77fff640 10 monclient: dump: 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 10 monclient: shutdown 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 1 -- 172.21.15.83:0/1336518635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 msgr2=0x7f0e64013ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:29.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.509+0000 7f0e8797e640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e64013ca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0e80146b40 tx=0x7f0e7802c440 comp rx=0 tx=0).stop 2024-03-25T01:59:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 1 -- 172.21.15.83:0/1336518635 shutdown_connections 2024-03-25T01:59:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e64013ca0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 1 --2- 172.21.15.83:0/1336518635 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64009f20 0x7f0e64005ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 1 -- 172.21.15.83:0/1336518635 >> 172.21.15.83:0/1336518635 conn(0x7f0e800b89b0 msgr2=0x7f0e800b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 1 -- 172.21.15.83:0/1336518635 shutdown_connections 2024-03-25T01:59:29.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 1 -- 172.21.15.83:0/1336518635 wait complete. 2024-03-25T01:59:29.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 10 monclient: build_initial_monmap 2024-03-25T01:59:29.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.510+0000 7f0e8797e640 10 monclient: monmap: 2024-03-25T01:59:29.515 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 librados: starting msgr at 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 librados: starting objecter 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 Processor -- start 2024-03-25T01:59:29.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 -- start start 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 librados: setting wanted keys 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 librados: calling monclient init 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient: init 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e64009f20 0x7f0e800acf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): picked mon.noname-a con 0x7f0e64009f20 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64013450 0x7f0e800ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e64013450 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e800b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): picked mon.noname-b con 0x7f0e64013830 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:29.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0e80147070 con 0x7f0e64009f20 2024-03-25T01:59:29.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:29.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e80146ef0 con 0x7f0e64013830 2024-03-25T01:59:29.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:29.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f0e801471f0 con 0x7f0e64013450 2024-03-25T01:59:29.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:29.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.511+0000 7f0e8797e640 10 monclient(hunting): authenticate will time out at 1288.700708s 2024-03-25T01:59:29.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.547+0000 7f0e8571a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e64009f20 0x7f0e800acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:29.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.548+0000 7f0e8571a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e64009f20 0x7f0e800acf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33162/0 (socket says 172.21.15.83:33162) 2024-03-25T01:59:29.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.548+0000 7f0e8571a640 1 -- 172.21.15.83:0/2235131909 learned_addr learned my addr 172.21.15.83:0/2235131909 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:29.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.548+0000 7f0e8571a640 10 monclient(hunting): get_auth_request con 0x7f0e64009f20 auth_method 0 2024-03-25T01:59:29.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.548+0000 7f0e8571a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:29.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.548+0000 7f0e8571a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:29.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.548+0000 7f0e8571a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:29.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.550+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e800b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:29.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.552+0000 7f0e85f1b640 10 monclient(hunting): get_auth_request con 0x7f0e64013830 auth_method 0 2024-03-25T01:59:29.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.552+0000 7f0e85f1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:29.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.552+0000 7f0e85f1b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:29.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.552+0000 7f0e85f1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:29.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.555+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:29.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.555+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:29.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.555+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 1 -- 172.21.15.83:0/2235131909 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64013450 msgr2=0x7f0e800ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64013450 0x7f0e800ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 1 -- 172.21.15.83:0/2235131909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e64009f20 msgr2=0x7f0e800acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e64009f20 0x7f0e800acf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 10 monclient: _finish_hunting 0 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 1 monclient: found mon.noname-b 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T01:59:29.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.559+0000 7f0e85f1b640 1 -- 172.21.15.83:0/2235131909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e800af150 con 0x7f0e64013830 2024-03-25T01:59:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.562+0000 7f0e85f1b640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e800b1c30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400f710 tx=0x7f0e78005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.562+0000 7f0e75ffb640 10 client.?.objecter ms_handle_connect 0x7f0e64013830 2024-03-25T01:59:29.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.563+0000 7f0e75ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 1 -- 172.21.15.83:0/2235131909 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78039040 con 0x7f0e64013830 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: dump: 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:29.598 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: _finish_auth 0 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: _check_auth_tickets 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 1 -- 172.21.15.83:0/2235131909 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e78031510 con 0x7f0e64013830 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 5 monclient: authenticate success, global_id 4114 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 10 monclient: _renew_subs 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8013a820 con 0x7f0e64013830 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 1 -- 172.21.15.83:0/2235131909 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78033be0 con 0x7f0e64013830 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:29.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e75ffb640 10 monclient: dump: 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e76ffd640 4 set_mon_vals no callback set 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 10 monclient: _renew_subs 2024-03-25T01:59:29.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:29.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e800b2350 con 0x7f0e64013830 2024-03-25T01:59:29.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.596+0000 7f0e8797e640 1 librados: init done 2024-03-25T01:59:29.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.597+0000 7f0e8797e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:29.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.597+0000 7f0e8797e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:29.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.597+0000 7f0e8797e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:29.601 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.597+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8004f940 con 0x7f0e64013830 2024-03-25T01:59:29.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.789+0000 7f0e75ffb640 1 -- 172.21.15.83:0/2235131909 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0e78047020 con 0x7f0e64013830 2024-03-25T01:59:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.795+0000 7f0e75ffb640 1 -- 172.21.15.83:0/2235131909 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0e7803ea30 con 0x7f0e64013830 2024-03-25T01:59:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.795+0000 7f0e75ffb640 10 client.4114.objecter ms_dispatch 0x7f0e80000b90 osd_map(1..1 src has 1..1) 2024-03-25T01:59:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.795+0000 7f0e75ffb640 3 client.4114.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T01:59:29.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.795+0000 7f0e75ffb640 3 client.4114.objecter handle_osd_map decoding full epoch 1 2024-03-25T01:59:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.795+0000 7f0e75ffb640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-25T01:59:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.796+0000 7f0e75ffb640 1 -- 172.21.15.83:0/2235131909 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0e7804e020 con 0x7f0e64013830 2024-03-25T01:59:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.796+0000 7f0e75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:29.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.796+0000 7f0e75ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:29.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.904+0000 7f0e8797e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T01:59:29.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.904+0000 7f0e8797e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T01:59:29.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.904+0000 7f0e8797e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:29.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:29.904+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f0e80147520 con 0x7f0e64013830 2024-03-25T01:59:30.073 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.072+0000 7f6ff8c3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T01:59:30.128 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.127+0000 7f6ff8c3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T01:59:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.133+0000 7f0e75ffb640 1 -- 172.21.15.83:0/2235131909 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f0e780316c0 con 0x7f0e64013830 2024-03-25T01:59:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.133+0000 7f0e75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T01:59:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.133+0000 7f0e75ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T01:59:30.134 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 10 librados: watch_flush enter 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 10 librados: watch_flush exit 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 10 client.4114.objecter successfully canceled tick 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 10 monclient: shutdown 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 msgr2=0x7f0e800b1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e800b1c30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0e6400f710 tx=0x7f0e78005dd0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 shutdown_connections 2024-03-25T01:59:30.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e64013830 0x7f0e800b1c30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f0e64013450 0x7f0e800ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 --2- 172.21.15.83:0/2235131909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0e64009f20 0x7f0e800acf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 >> 172.21.15.83:0/2235131909 conn(0x7f0e800b89b0 msgr2=0x7f0e800b78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 shutdown_connections 2024-03-25T01:59:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 -- 172.21.15.83:0/2235131909 wait complete. 2024-03-25T01:59:30.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:30.134+0000 7f0e8797e640 1 librados: shutdown 2024-03-25T01:59:30.146 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T01:59:30.146 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T01:59:30.146 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:30.147 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T01:59:30.174 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:30.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T01:59:30.241 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:30.241 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T01:59:30.253 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:30.252+0000 7f85a6012040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T01:59:30.279 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T01:59:30.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T01:59:30.319 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:30.318+0000 7f85a6012040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T01:59:30.345 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:30.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T01:59:30.373 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:30.373 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T01:59:30.441 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:30.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T01:59:30.508 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-25T01:59:30.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T01:59:30.576 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new c3639213-7917-42d1-9609-bdc85e49f7bf 0 2024-03-25T01:59:30.662 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.661+0000 7f6ff8c3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.715+0000 7f2132f82640 10 monclient: get_monmap_and_config 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.715+0000 7f2132f82640 10 monclient: build_initial_monmap 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.716+0000 7f2132f82640 10 monclient: monmap: 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:30.716846+0000 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:30.716846+0000 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:30.717 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:30.718 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:30.718 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:30.718 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:30.718 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:30.718 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.716+0000 7f2132f82640 1 Processor -- start 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.716+0000 7f2132f82640 1 -- start start 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.716+0000 7f2132f82640 10 monclient: init 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f212c0b1900 0x7f212c0b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient(hunting): picked mon.noname-a con 0x7f212c0b1900 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f212c13e130 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f212c1447d0 0x7f212c144c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient(hunting): picked mon.noname-c con 0x7f212c1447d0 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f212c0b0f70 con 0x7f212c13e130 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f212c0b0d20 con 0x7f212c0b1900 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f212c0a9aa0 con 0x7f212c1447d0 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f2132f82640 10 monclient(hunting): authenticate will time out at 1293.000094s 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.717+0000 7f213151f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f212c1447d0 0x7f212c144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f2130d1e640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f212c0b1900 0x7f212c0b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f212c1447d0 0x7f212c144c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33536/0 (socket says 172.21.15.153:33536) 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 1 -- 172.21.15.153:0/4145420438 learned_addr learned my addr 172.21.15.153:0/4145420438 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 10 monclient(hunting): get_auth_request con 0x7f212c1447d0 auth_method 0 2024-03-25T01:59:30.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f2130d1e640 10 monclient(hunting): get_auth_request con 0x7f212c0b1900 auth_method 0 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f2130d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f2130d1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f2130d1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:30.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:30.723 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.718+0000 7f6ff8c3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T01:59:30.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.719+0000 7f2130d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:30.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.719+0000 7f2130d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.719+0000 7f2130d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 1 -- 172.21.15.153:0/4145420438 >> v2:172.21.15.83:3300/0 conn(0x7f212c0b1900 msgr2=0x7f212c0b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 1 --2- 172.21.15.153:0/4145420438 >> v2:172.21.15.83:3300/0 conn(0x7f212c0b1900 0x7f212c0b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 1 -- 172.21.15.153:0/4145420438 >> v1:172.21.15.83:6789/0 conn(0x7f212c13e130 legacy=0x7f212c13e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 10 monclient: _finish_hunting 0 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 1 monclient: found mon.noname-c 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.720+0000 7f213151f640 1 -- 172.21.15.153:0/4145420438 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c0a9d80 con 0x7f212c1447d0 2024-03-25T01:59:30.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.722+0000 7f213151f640 1 --2- 172.21.15.153:0/4145420438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c144c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f212c0b0c90 tx=0x7f212402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeae2ea1a038df8d server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 1 -- 172.21.15.153:0/4145420438 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212402ca80 con 0x7f212c1447d0 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 10 monclient: dump: 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:30.728 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 10 monclient: _finish_auth 0 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 10 monclient: _check_auth_tickets 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.725+0000 7f212b7fe640 1 -- 172.21.15.153:0/4145420438 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f212402cc20 con 0x7f212c1447d0 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 5 monclient: authenticate success, global_id 4117 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f212b7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 4 set_mon_vals no callback set 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f212b7fe640 1 -- 172.21.15.153:0/4145420438 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21240343f0 con 0x7f212c1447d0 2024-03-25T01:59:30.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f212b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f212b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f212b7fe640 10 monclient: dump: 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 10 monclient: shutdown 2024-03-25T01:59:30.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 -- 172.21.15.153:0/4145420438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 msgr2=0x7f212c144c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 --2- 172.21.15.153:0/4145420438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c144c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f212c0b0c90 tx=0x7f212402bee0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 -- 172.21.15.153:0/4145420438 shutdown_connections 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 --2- 172.21.15.153:0/4145420438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c144c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 --2- 172.21.15.153:0/4145420438 >> v2:172.21.15.83:3300/0 conn(0x7f212c0b1900 0x7f212c0b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 -- 172.21.15.153:0/4145420438 >> 172.21.15.153:0/4145420438 conn(0x7f212c067d40 msgr2=0x7f212c0b68c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 -- 172.21.15.153:0/4145420438 shutdown_connections 2024-03-25T01:59:30.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.726+0000 7f2132f82640 1 -- 172.21.15.153:0/4145420438 wait complete. 2024-03-25T01:59:30.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 10 monclient: build_initial_monmap 2024-03-25T01:59:30.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 10 monclient: monmap: 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 1 librados: starting msgr at 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 1 librados: starting objecter 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 1 Processor -- start 2024-03-25T01:59:30.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 1 -- start start 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 1 librados: setting wanted keys 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 1 librados: calling monclient init 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.727+0000 7f2132f82640 10 monclient: init 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f212c1435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): picked mon.noname-a con 0x7f212c0b1900 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 0x7f212c06cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): picked mon.noname-c con 0x7f212c13e130 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c06d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): picked mon.noname-b con 0x7f212c1447d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f212c14b860 con 0x7f212c0b1900 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f212c14b9e0 con 0x7f212c1447d0 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f212c14bb60 con 0x7f212c13e130 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.728+0000 7f2132f82640 10 monclient(hunting): authenticate will time out at 1293.011296s 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c06d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f212c1435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c06d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33552/0 (socket says 172.21.15.153:33552) 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 learned_addr learned my addr 172.21.15.153:0/761896986 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f212bfff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 0x7f212c06cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 10 monclient(hunting): get_auth_request con 0x7f212c1447d0 auth_method 0 2024-03-25T01:59:30.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 10 monclient(hunting): get_auth_request con 0x7f212c0b1900 auth_method 0 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f212bfff640 10 monclient(hunting): get_auth_request con 0x7f212c13e130 auth_method 0 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f212bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f212bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f212bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:30.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.729+0000 7f2130d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.730+0000 7f212bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.730+0000 7f212bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.730+0000 7f212bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 msgr2=0x7f212c06cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 0x7f212c06cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 msgr2=0x7f212c1435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f212c1435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 10 monclient: _finish_hunting 0 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 1 monclient: found mon.noname-b 2024-03-25T01:59:30.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:30.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T01:59:30.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212c06d490 con 0x7f212c1447d0 2024-03-25T01:59:30.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.732+0000 7f2130d1e640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f212c1435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T01:59:30.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.734+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c06d9d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2124036040 tx=0x7f2124004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:30.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.734+0000 7f21297fa640 10 client.?.objecter ms_handle_connect 0x7f212c1447d0 2024-03-25T01:59:30.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.734+0000 7f21297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 msgr2=0x7f212c06d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 msgr2=0x7f212c06d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c06d9d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2124036040 tx=0x7f2124004a40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c06d9d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2124036040 tx=0x7f2124004a40 comp rx=0 tx=0).stop 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 msgr2=0x7f212c06d9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f212c1447d0 0x7f212c06d9d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T01:59:30.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2110004e60 0x7f21100072b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2110004e60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 0x7f2110009e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient(hunting): picked mon.noname-c con 0x7f212c13e130 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f211000c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient(hunting): picked mon.noname-a con 0x7f212c0b1900 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f211000d2c0 con 0x7f212c0b1900 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f211000d440 con 0x7f2110004e60 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f211000d640 con 0x7f212c13e130 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f21297fa640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:30.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.736+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f211000c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f213151f640 10 monclient(hunting): get_auth_request con 0x7f212c0b1900 auth_method 0 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f213151f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f213151f640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f213151f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f2130d1e640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2110004e60 0x7f21100072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.737+0000 7f213151f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.738+0000 7f212bfff640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 0x7f2110009e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.738+0000 7f212bfff640 10 monclient(hunting): get_auth_request con 0x7f212c13e130 auth_method 0 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.738+0000 7f212bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:30.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.738+0000 7f212bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.738+0000 7f212bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f212bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f212bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f212bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 msgr2=0x7f2110009e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 0x7f2110009e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2110004e60 msgr2=0x7f21100072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2110004e60 0x7f21100072b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 10 monclient: _finish_hunting 0 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 1 monclient: found mon.noname-a 2024-03-25T01:59:30.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:30.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:30.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.739+0000 7f213151f640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211000d900 con 0x7f212c0b1900 2024-03-25T01:59:30.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.747+0000 7f21297fa640 10 client.?.objecter ms_handle_connect 0x7f212c0b1900 2024-03-25T01:59:30.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.747+0000 7f21297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:30.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.747+0000 7f213151f640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f211000c8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2124041000 tx=0x7f2124030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:30.767 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.767+0000 7f6ff8c3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T01:59:30.821 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.820+0000 7f6ff8c3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T01:59:30.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124043020 con 0x7f212c0b1900 2024-03-25T01:59:30.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:30.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:30.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: dump: 2024-03-25T01:59:30.836 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:30.836 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:30.836 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.836 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.836 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:30.836 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:30.837 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: _finish_auth 0 2024-03-25T01:59:30.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: _check_auth_tickets 2024-03-25T01:59:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f212403e450 con 0x7f212c0b1900 2024-03-25T01:59:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212403e620 con 0x7f212c0b1900 2024-03-25T01:59:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 5 monclient: authenticate success, global_id 4120 2024-03-25T01:59:30.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 10 monclient: _renew_subs 2024-03-25T01:59:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212c145d70 con 0x7f212c0b1900 2024-03-25T01:59:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f21297fa640 10 monclient: dump: 2024-03-25T01:59:30.839 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:30.840 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:30.841 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f212affd640 4 set_mon_vals no callback set 2024-03-25T01:59:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 10 monclient: _renew_subs 2024-03-25T01:59:30.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f212c1dbc10 con 0x7f212c0b1900 2024-03-25T01:59:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.832+0000 7f2132f82640 1 librados: init done 2024-03-25T01:59:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.833+0000 7f2132f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.833+0000 7f2132f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.833+0000 7f2132f82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:30.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.833+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f212c04f940 con 0x7f212c0b1900 2024-03-25T01:59:30.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.849+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2124046020 con 0x7f212c0b1900 2024-03-25T01:59:30.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.849+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f2124003d00 con 0x7f212c0b1900 2024-03-25T01:59:30.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.849+0000 7f21297fa640 10 client.4120.objecter ms_dispatch 0x7f212c000b90 osd_map(2..2 src has 1..2) 2024-03-25T01:59:30.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.849+0000 7f21297fa640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T01:59:30.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.849+0000 7f21297fa640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-25T01:59:30.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.849+0000 7f21297fa640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-25T01:59:30.852 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:30.851+0000 7f85a6012040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T01:59:30.864 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.863+0000 7f6ff8c3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T01:59:30.906 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:30.906+0000 7f85a6012040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T01:59:30.952 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:30.951+0000 7f85a6012040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T01:59:30.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.964+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f212402ca90 con 0x7f212c0b1900 2024-03-25T01:59:30.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.964+0000 7f21297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:30.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:30.964+0000 7f21297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:30.980 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:30.979+0000 7f6ff8c3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T01:59:31.006 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.005+0000 7f85a6012040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T01:59:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.036+0000 7f2132f82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c3639213-7917-42d1-9609-bdc85e49f7bf", "id": 0}] 2024-03-25T01:59:31.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.036+0000 7f2132f82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c3639213-7917-42d1-9609-bdc85e49f7bf", "id": 0}] 2024-03-25T01:59:31.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.036+0000 7f2132f82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:31.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.036+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c3639213-7917-42d1-9609-bdc85e49f7bf", "id": 0} v 0) -- 0x7f212c06d730 con 0x7f212c0b1900 2024-03-25T01:59:31.048 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.048+0000 7f85a6012040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T01:59:31.052 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.051+0000 7f6ff8c3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T01:59:31.097 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.096+0000 7f6ff8c3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T01:59:31.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.126+0000 7f21297fa640 1 -- 172.21.15.153:0/761896986 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c3639213-7917-42d1-9609-bdc85e49f7bf", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f212403e7c0 con 0x7f212c0b1900 2024-03-25T01:59:31.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.126+0000 7f21297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c3639213-7917-42d1-9609-bdc85e49f7bf", "id": 0}] 2024-03-25T01:59:31.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.126+0000 7f21297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:31.127 INFO:teuthology.orchestra.run.smithi153.stdout:0 2024-03-25T01:59:31.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 10 librados: watch_flush enter 2024-03-25T01:59:31.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 10 librados: watch_flush exit 2024-03-25T01:59:31.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-25T01:59:31.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 10 client.4120.objecter successfully canceled tick 2024-03-25T01:59:31.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 10 monclient: shutdown 2024-03-25T01:59:31.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:31.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 msgr2=0x7f211000c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f211000c8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2124041000 tx=0x7f2124030000 comp rx=0 tx=0).stop 2024-03-25T01:59:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 shutdown_connections 2024-03-25T01:59:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f212c0b1900 0x7f211000c8b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f212c13e130 0x7f2110009e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 --2- 172.21.15.153:0/761896986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2110004e60 0x7f21100072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 >> 172.21.15.153:0/761896986 conn(0x7f212c067d40 msgr2=0x7f212c074f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 shutdown_connections 2024-03-25T01:59:31.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 -- 172.21.15.153:0/761896986 wait complete. 2024-03-25T01:59:31.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.127+0000 7f2132f82640 1 librados: shutdown 2024-03-25T01:59:31.142 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 8c5bf12d-d066-4fe7-a189-11c1d757e4b0 1 2024-03-25T01:59:31.154 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.153+0000 7f6ff8c3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T01:59:31.171 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.171+0000 7f85a6012040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T01:59:31.204 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.204+0000 7f6ff8c3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.243+0000 7f83bed24640 10 monclient: get_monmap_and_config 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.243+0000 7f83bed24640 10 monclient: build_initial_monmap 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.243+0000 7f83bed24640 10 monclient: monmap: 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:31.243978+0000 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:31.243978+0000 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:31.245 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:31.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:31.245 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:31.245 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:31.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.243+0000 7f83bed24640 1 Processor -- start 2024-03-25T01:59:31.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.243+0000 7f83bed24640 1 -- start start 2024-03-25T01:59:31.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.243+0000 7f83bed24640 10 monclient: init 2024-03-25T01:59:31.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f83b80a9b20 0x7f83b8146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): picked mon.noname-a con 0x7f83b80a9b20 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): picked mon.noname-b con 0x7f83b80b3330 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f83b80a9f00 0x7f83b81408f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): picked mon.noname-c con 0x7f83b80a9f00 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:31.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f83b80abb00 con 0x7f83b80a9b20 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f83b80abde0 con 0x7f83b80a9f00 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f83b80b2e80 con 0x7f83b80b3330 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.244+0000 7f83bed24640 10 monclient(hunting): authenticate will time out at 1293.527203s 2024-03-25T01:59:31.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f83b80a9f00 0x7f83b81408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bcac0640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f83b80a9b20 0x7f83b8146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81403b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.153:60818/0 (socket says 172.21.15.153:60818) 2024-03-25T01:59:31.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 1 -- 172.21.15.153:0/1494530184 learned_addr learned my addr 172.21.15.153:0/1494530184 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:31.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 10 monclient(hunting): get_auth_request con 0x7f83b80b3330 auth_method 0 2024-03-25T01:59:31.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 10 monclient(hunting): get_auth_request con 0x7f83b80a9f00 auth_method 0 2024-03-25T01:59:31.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bcac0640 10 monclient(hunting): get_auth_request con 0x7f83b80a9b20 auth_method 0 2024-03-25T01:59:31.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bcac0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bcac0640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bcac0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83bd2c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.245+0000 7f83b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.246+0000 7f83bcac0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.246+0000 7f83bcac0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.246+0000 7f83bcac0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.254 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.248+0000 7f85a6012040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T01:59:31.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-03-25T01:59:31.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 1 -- 172.21.15.153:0/1494530184 >> v2:172.21.15.153:3300/0 conn(0x7f83b80a9f00 msgr2=0x7f83b81408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:31.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 1 --2- 172.21.15.153:0/1494530184 >> v2:172.21.15.153:3300/0 conn(0x7f83b80a9f00 0x7f83b81408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 1 -- 172.21.15.153:0/1494530184 >> v2:172.21.15.83:3300/0 conn(0x7f83b80a9b20 msgr2=0x7f83b8146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:31.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 1 --2- 172.21.15.153:0/1494530184 >> v2:172.21.15.83:3300/0 conn(0x7f83b80a9b20 0x7f83b8146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 10 monclient: _finish_hunting 0 2024-03-25T01:59:31.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 1 monclient: found mon.noname-b 2024-03-25T01:59:31.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:31.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T01:59:31.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.247+0000 7f83b7fff640 1 -- 172.21.15.153:0/1494530184 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b80aaef0 con 0x7f83b80b3330 2024-03-25T01:59:31.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.249+0000 7f83b7fff640 1 --2- 172.21.15.153:0/1494530184 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81403b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f83b81441e0 tx=0x7f83a402b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=876720c56996e006 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:31.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 1 -- 172.21.15.153:0/1494530184 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4038070 con 0x7f83b80b3330 2024-03-25T01:59:31.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:31.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: dump: 2024-03-25T01:59:31.263 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.263 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.263 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.263 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.263 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.264 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.264 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.264 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.264 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.264 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: _finish_auth 0 2024-03-25T01:59:31.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: _check_auth_tickets 2024-03-25T01:59:31.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:31.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 1 -- 172.21.15.153:0/1494530184 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a402f050 con 0x7f83b80b3330 2024-03-25T01:59:31.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:31.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 1 -- 172.21.15.153:0/1494530184 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4031970 con 0x7f83b80b3330 2024-03-25T01:59:31.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83bed24640 5 monclient: authenticate success, global_id 4106 2024-03-25T01:59:31.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83bed24640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:31.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83bed24640 4 set_mon_vals no callback set 2024-03-25T01:59:31.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:31.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.254+0000 7f83b77fe640 10 monclient: dump: 2024-03-25T01:59:31.266 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.267 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.267 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.267 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.267 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.267 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.267 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.268 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.268 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.268 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 10 monclient: shutdown 2024-03-25T01:59:31.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:31.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 -- 172.21.15.153:0/1494530184 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 msgr2=0x7f83b81403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 --2- 172.21.15.153:0/1494530184 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81403b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f83b81441e0 tx=0x7f83a402b7f0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 -- 172.21.15.153:0/1494530184 shutdown_connections 2024-03-25T01:59:31.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 --2- 172.21.15.153:0/1494530184 >> v2:172.21.15.153:3300/0 conn(0x7f83b80a9f00 0x7f83b81408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 --2- 172.21.15.153:0/1494530184 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81403b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 --2- 172.21.15.153:0/1494530184 >> v2:172.21.15.83:3300/0 conn(0x7f83b80a9b20 0x7f83b8146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 -- 172.21.15.153:0/1494530184 >> 172.21.15.153:0/1494530184 conn(0x7f83b80b89b0 msgr2=0x7f83b80b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:31.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 -- 172.21.15.153:0/1494530184 shutdown_connections 2024-03-25T01:59:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.255+0000 7f83bed24640 1 -- 172.21.15.153:0/1494530184 wait complete. 2024-03-25T01:59:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 10 monclient: build_initial_monmap 2024-03-25T01:59:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 10 monclient: monmap: 2024-03-25T01:59:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:31.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 1 librados: starting msgr at 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 1 librados: starting objecter 2024-03-25T01:59:31.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 1 Processor -- start 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 1 -- start start 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 1 librados: setting wanted keys 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 1 librados: calling monclient init 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.256+0000 7f83bed24640 10 monclient: init 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 0x7f83b81cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): picked mon.noname-a con 0x7f83b80a9b20 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:31.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83b80a9f00 0x7f83b81d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): picked mon.noname-b con 0x7f83b80a9f00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81d91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): picked mon.noname-c con 0x7f83b80b3330 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f83b8147430 con 0x7f83b80a9b20 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f83b8143140 con 0x7f83b80a9f00 2024-03-25T01:59:31.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f83b81ccc00 con 0x7f83b80b3330 2024-03-25T01:59:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bed24640 10 monclient(hunting): authenticate will time out at 1293.540075s 2024-03-25T01:59:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83bcac0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 0x7f83b81cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.257+0000 7f83b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83b80a9f00 0x7f83b81d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83b80a9f00 0x7f83b81d8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33574/0 (socket says 172.21.15.153:33574) 2024-03-25T01:59:31.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 1 -- 172.21.15.153:0/3506943410 learned_addr learned my addr 172.21.15.153:0/3506943410 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 10 monclient(hunting): get_auth_request con 0x7f83b80a9f00 auth_method 0 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83bcac0640 1 -- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 msgr2=0x7f83b81cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83bcac0640 1 -- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 msgr2=0x7f83b81cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.258+0000 7f83bcac0640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 0x7f83b81cc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:31.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.259+0000 7f83bcac0640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 0x7f83b81cc6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:31.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 1 -- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 msgr2=0x7f83b81d91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81d91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 1 -- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 msgr2=0x7f83b81cc6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 0x7f83b81cc6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 10 monclient: _finish_hunting 0 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 1 monclient: found mon.noname-b 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.261+0000 7f83b7fff640 1 -- 172.21.15.153:0/3506943410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b81cebe0 con 0x7f83b80a9f00 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.263+0000 7f83b7fff640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83b80a9f00 0x7f83b81d8c70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f83a402be00 tx=0x7f83a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.263+0000 7f83b57fa640 10 client.?.objecter ms_handle_connect 0x7f83b80a9f00 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.263+0000 7f83b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:31.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 1 -- 172.21.15.153:0/3506943410 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4042020 con 0x7f83b80a9f00 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: dump: 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: _finish_auth 0 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: _check_auth_tickets 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 1 -- 172.21.15.153:0/3506943410 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a403d660 con 0x7f83b80a9f00 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 1 -- 172.21.15.153:0/3506943410 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4038040 con 0x7f83b80a9f00 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b57fa640 10 monclient: dump: 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.280 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 5 monclient: authenticate success, global_id 4126 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83b6ffd640 4 set_mon_vals no callback set 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 10 monclient: _renew_subs 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b81d96f0 con 0x7f83b80a9f00 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 10 monclient: _renew_subs 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b81cd530 con 0x7f83b80a9f00 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.267+0000 7f83bed24640 1 librados: init done 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.268+0000 7f83bed24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:31.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.268+0000 7f83bed24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.268+0000 7f83bed24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.268+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b81ccd90 con 0x7f83b80a9f00 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.270+0000 7f83b57fa640 1 -- 172.21.15.153:0/3506943410 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f83a4045020 con 0x7f83b80a9f00 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.270+0000 7f83b57fa640 1 -- 172.21.15.153:0/3506943410 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f83a4051c30 con 0x7f83b80a9f00 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.270+0000 7f83b57fa640 10 client.4126.objecter ms_dispatch 0x7f83b8000b90 osd_map(3..3 src has 1..3) 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.270+0000 7f83b57fa640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.270+0000 7f83b57fa640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T01:59:31.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.270+0000 7f83b57fa640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T01:59:31.297 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.295+0000 7f85a6012040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T01:59:31.351 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.350+0000 7f85a6012040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T01:59:31.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.370+0000 7f83b57fa640 1 -- 172.21.15.153:0/3506943410 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f83b81ccd90 con 0x7f83b80a9f00 2024-03-25T01:59:31.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.370+0000 7f83b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:31.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.370+0000 7f83b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:31.401 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.399+0000 7f85a6012040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T01:59:31.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.444+0000 7f83bed24640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8c5bf12d-d066-4fe7-a189-11c1d757e4b0", "id": 1}] 2024-03-25T01:59:31.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.444+0000 7f83bed24640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8c5bf12d-d066-4fe7-a189-11c1d757e4b0", "id": 1}] 2024-03-25T01:59:31.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.444+0000 7f83bed24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.444+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8c5bf12d-d066-4fe7-a189-11c1d757e4b0", "id": 1} v 0) -- 0x7f83b804f940 con 0x7f83b80a9f00 2024-03-25T01:59:31.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.561+0000 7f83b57fa640 1 -- 172.21.15.153:0/3506943410 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8c5bf12d-d066-4fe7-a189-11c1d757e4b0", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f83a404c020 con 0x7f83b80a9f00 2024-03-25T01:59:31.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.561+0000 7f83b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8c5bf12d-d066-4fe7-a189-11c1d757e4b0", "id": 1}] 2024-03-25T01:59:31.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.561+0000 7f83b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:31.563 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 10 librados: watch_flush enter 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 10 librados: watch_flush exit 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 10 client.4126.objecter successfully canceled tick 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 10 monclient: shutdown 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83b80a9f00 msgr2=0x7f83b81d8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83b80a9f00 0x7f83b81d8c70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f83a402be00 tx=0x7f83a40046c0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 shutdown_connections 2024-03-25T01:59:31.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f83b80b3330 0x7f83b81d91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f83b80a9f00 0x7f83b81d8c70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 --2- 172.21.15.153:0/3506943410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b80a9b20 0x7f83b81cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 >> 172.21.15.153:0/3506943410 conn(0x7f83b80b89b0 msgr2=0x7f83b80b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 shutdown_connections 2024-03-25T01:59:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 -- 172.21.15.153:0/3506943410 wait complete. 2024-03-25T01:59:31.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.562+0000 7f83bed24640 1 librados: shutdown 2024-03-25T01:59:31.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 19c0b51e-bebb-48b0-a930-bb1c181de85c 2 2024-03-25T01:59:31.600 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.598+0000 7f6ff8c3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T01:59:31.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.699+0000 7f6ba20c5640 10 monclient: get_monmap_and_config 2024-03-25T01:59:31.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.699+0000 7f6ba20c5640 10 monclient: build_initial_monmap 2024-03-25T01:59:31.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.699+0000 7f6ba20c5640 10 monclient: monmap: 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:31.699938+0000 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:31.699938+0000 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:31.701 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:31.702 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:31.702 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.699+0000 7f6ba20c5640 1 Processor -- start 2024-03-25T01:59:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.699+0000 7f6ba20c5640 1 -- start start 2024-03-25T01:59:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.699+0000 7f6ba20c5640 10 monclient: init 2024-03-25T01:59:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-03-25T01:59:31.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b9c0a9b20 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:31.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6b9c0b3330 0x7f6b9c1403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient(hunting): picked mon.noname-c con 0x7f6b9c0b3330 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:31.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b9c0a9f00 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:31.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9c0abb00 con 0x7f6b9c0a9b20 2024-03-25T01:59:31.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9c0abde0 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f6b9c0b2e80 con 0x7f6b9c0b3330 2024-03-25T01:59:31.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:31.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.700+0000 7f6ba20c5640 10 monclient(hunting): authenticate will time out at 1293.983250s 2024-03-25T01:59:31.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6b9c0b3330 0x7f6b9c1403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9b7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f6b9c0a9b20 0x7f6b9c146ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.153:46526/0 (socket says 172.21.15.153:46526) 2024-03-25T01:59:31.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9b7fe640 1 -- 172.21.15.153:0/3687017549 learned_addr learned my addr 172.21.15.153:0/3687017549 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T01:59:31.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 10 monclient(hunting): get_auth_request con 0x7f6b9c0b3330 auth_method 0 2024-03-25T01:59:31.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.701+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 1 -- 172.21.15.153:0/3687017549 >> v1:172.21.15.153:6789/0 conn(0x7f6b9c0a9f00 legacy=0x7f6b9c14d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 1 -- 172.21.15.153:0/3687017549 >> v1:172.21.15.83:6789/0 conn(0x7f6b9c0a9b20 legacy=0x7f6b9c146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 10 monclient: _finish_hunting 0 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 1 monclient: found mon.noname-c 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9a7fc640 1 -- 172.21.15.153:0/3687017549 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 665012853 0 0) 0x7f6b9c0abb00 con 0x7f6b9c0a9b20 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9affd640 1 -- 172.21.15.153:0/3687017549 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c1473f0 con 0x7f6b9c0b3330 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.703+0000 7f6b9a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-03-25T01:59:31.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.706+0000 7f6b9affd640 1 --2- 172.21.15.153:0/3687017549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0b3330 0x7f6b9c1403b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b900097a0 tx=0x7f6b9002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62df8433f54480b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6b9a7fc640 1 -- 172.21.15.153:0/3687017549 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90038070 con 0x7f6b9c0b3330 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6b9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6b9a7fc640 10 monclient: dump: 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.712 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6b9a7fc640 10 monclient: _finish_auth 0 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6b9a7fc640 10 monclient: _check_auth_tickets 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6b9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6ba20c5640 5 monclient: authenticate success, global_id 4129 2024-03-25T01:59:31.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.710+0000 7f6ba20c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6b9a7fc640 1 -- 172.21.15.153:0/3687017549 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b90031700 con 0x7f6b9c0b3330 2024-03-25T01:59:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6b9a7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6b9a7fc640 1 -- 172.21.15.153:0/3687017549 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90031ad0 con 0x7f6b9c0b3330 2024-03-25T01:59:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6b9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6b9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6b9a7fc640 10 monclient: dump: 2024-03-25T01:59:31.715 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6ba20c5640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.712+0000 7f6ba20c5640 4 set_mon_vals no callback set 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 10 monclient: shutdown 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 1 -- 172.21.15.153:0/3687017549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0b3330 msgr2=0x7f6b9c1403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 1 --2- 172.21.15.153:0/3687017549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0b3330 0x7f6b9c1403b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6b900097a0 tx=0x7f6b9002b7a0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 1 -- 172.21.15.153:0/3687017549 shutdown_connections 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 1 --2- 172.21.15.153:0/3687017549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0b3330 0x7f6b9c1403b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 1 -- 172.21.15.153:0/3687017549 >> 172.21.15.153:0/3687017549 conn(0x7f6b9c0b89b0 msgr2=0x7f6b9c0b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 1 -- 172.21.15.153:0/3687017549 shutdown_connections 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 1 -- 172.21.15.153:0/3687017549 wait complete. 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 10 monclient: build_initial_monmap 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.713+0000 7f6ba20c5640 10 monclient: monmap: 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:31.717 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.714+0000 7f6ba20c5640 1 librados: starting msgr at 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.714+0000 7f6ba20c5640 1 librados: starting objecter 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.714+0000 7f6ba20c5640 1 Processor -- start 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.714+0000 7f6ba20c5640 1 -- start start 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.714+0000 7f6ba20c5640 1 librados: setting wanted keys 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.714+0000 7f6ba20c5640 1 librados: calling monclient init 2024-03-25T01:59:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.714+0000 7f6ba20c5640 10 monclient: init 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b9c0a9b20 0x7f6b9c145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): picked mon.noname-a con 0x7f6b9c0a9b20 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0a9f00 0x7f6b9c1d8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): picked mon.noname-b con 0x7f6b9c0a9f00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b9c0b3330 0x7f6b9c1d8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:31.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): picked mon.noname-c con 0x7f6b9c0b3330 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c140a70 con 0x7f6b9c0a9b20 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c1408f0 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c140bf0 con 0x7f6b9c0b3330 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6ba20c5640 10 monclient(hunting): authenticate will time out at 1293.997806s 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0a9f00 0x7f6b9c1d8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b9c0a9b20 0x7f6b9c145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9bfff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b9c0b3330 0x7f6b9c1d8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0a9f00 0x7f6b9c1d8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33590/0 (socket says 172.21.15.153:33590) 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9affd640 1 -- 172.21.15.153:0/4211648274 learned_addr learned my addr 172.21.15.153:0/4211648274 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:31.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9affd640 10 monclient(hunting): get_auth_request con 0x7f6b9c0a9f00 auth_method 0 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.715+0000 7f6b9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9bfff640 10 monclient(hunting): get_auth_request con 0x7f6b9c0b3330 auth_method 0 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9b7fe640 10 monclient(hunting): get_auth_request con 0x7f6b9c0a9b20 auth_method 0 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.716+0000 7f6b9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 1 -- 172.21.15.153:0/4211648274 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b9c0b3330 msgr2=0x7f6b9c1d8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 1 --2- 172.21.15.153:0/4211648274 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b9c0b3330 0x7f6b9c1d8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 1 -- 172.21.15.153:0/4211648274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b9c0a9b20 msgr2=0x7f6b9c145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 1 --2- 172.21.15.153:0/4211648274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b9c0a9b20 0x7f6b9c145720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 10 monclient: _finish_hunting 0 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 1 monclient: found mon.noname-b 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.719+0000 7f6b9affd640 1 -- 172.21.15.153:0/4211648274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c0acc10 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.721+0000 7f6b9affd640 1 --2- 172.21.15.153:0/4211648274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0a9f00 0x7f6b9c1d8210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b900098c0 tx=0x7f6b900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:31.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.721+0000 7f6b83fff640 10 client.?.objecter ms_handle_connect 0x7f6b9c0a9f00 2024-03-25T01:59:31.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.721+0000 7f6b83fff640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:31.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 1 -- 172.21.15.153:0/4211648274 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90043070 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:31.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: dump: 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: _finish_auth 0 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: _check_auth_tickets 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 1 -- 172.21.15.153:0/4211648274 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b90033a30 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 1 -- 172.21.15.153:0/4211648274 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b90038040 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 5 monclient: authenticate success, global_id 4132 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 10 monclient: _renew_subs 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c0ace80 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b83fff640 10 monclient: dump: 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:31.728 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6b99ffb640 4 set_mon_vals no callback set 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 10 monclient: _renew_subs 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c143940 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.725+0000 7f6ba20c5640 1 librados: init done 2024-03-25T01:59:31.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.726+0000 7f6ba20c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:31.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.726+0000 7f6ba20c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:31.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.726+0000 7f6ba20c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.726+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c04f940 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.744 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.743+0000 7f6ff8c3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T01:59:31.789 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.787+0000 7f85a6012040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T01:59:31.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 1 -- 172.21.15.153:0/4211648274 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6b90046070 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 1 -- 172.21.15.153:0/4211648274 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f6b9003f460 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 10 client.4132.objecter ms_dispatch 0x7f6b9c000b90 osd_map(4..4 src has 1..4) 2024-03-25T01:59:31.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T01:59:31.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-25T01:59:31.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T01:59:31.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 1 -- 172.21.15.153:0/4211648274 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6b90030070 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.825+0000 7f6b83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:31.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.826+0000 7f6b83fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:31.859 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.858+0000 7f6ff8c3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T01:59:31.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.898+0000 7f6ba20c5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "19c0b51e-bebb-48b0-a930-bb1c181de85c", "id": 2}] 2024-03-25T01:59:31.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.898+0000 7f6ba20c5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "19c0b51e-bebb-48b0-a930-bb1c181de85c", "id": 2}] 2024-03-25T01:59:31.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.898+0000 7f6ba20c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:31.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.898+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "19c0b51e-bebb-48b0-a930-bb1c181de85c", "id": 2} v 0) -- 0x7f6b9c143270 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.907 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:31.906+0000 7f6ff8c3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T01:59:31.922 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:31.921+0000 7f85a6012040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T01:59:31.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.959+0000 7f6b83fff640 1 -- 172.21.15.153:0/4211648274 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "19c0b51e-bebb-48b0-a930-bb1c181de85c", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f6b90033e20 con 0x7f6b9c0a9f00 2024-03-25T01:59:31.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.959+0000 7f6b83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "19c0b51e-bebb-48b0-a930-bb1c181de85c", "id": 2}] 2024-03-25T01:59:31.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.959+0000 7f6b83fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:31.961 INFO:teuthology.orchestra.run.smithi153.stdout:2 2024-03-25T01:59:31.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 10 librados: watch_flush enter 2024-03-25T01:59:31.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 10 librados: watch_flush exit 2024-03-25T01:59:31.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T01:59:31.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 10 client.4132.objecter successfully canceled tick 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 10 monclient: shutdown 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0a9f00 msgr2=0x7f6b9c1d8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 --2- 172.21.15.153:0/4211648274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0a9f00 0x7f6b9c1d8210 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6b900098c0 tx=0x7f6b900046c0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 shutdown_connections 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 --2- 172.21.15.153:0/4211648274 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b9c0b3330 0x7f6b9c1d8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 --2- 172.21.15.153:0/4211648274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b9c0a9f00 0x7f6b9c1d8210 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 --2- 172.21.15.153:0/4211648274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b9c0a9b20 0x7f6b9c145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 >> 172.21.15.153:0/4211648274 conn(0x7f6b9c0b89b0 msgr2=0x7f6b9c13b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:31.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 shutdown_connections 2024-03-25T01:59:31.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 -- 172.21.15.153:0/4211648274 wait complete. 2024-03-25T01:59:31.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:31.960+0000 7f6ba20c5640 1 librados: shutdown 2024-03-25T01:59:31.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 415b88b9-7d5a-4162-a686-902197aca9fb 3 2024-03-25T01:59:32.030 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:32.029+0000 7f6ff8c3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T01:59:32.045 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:32.043+0000 7f85a6012040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T01:59:32.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.073+0000 7f06b87bd640 10 monclient: get_monmap_and_config 2024-03-25T01:59:32.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.073+0000 7f06b87bd640 10 monclient: build_initial_monmap 2024-03-25T01:59:32.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.073+0000 7f06b87bd640 10 monclient: monmap: 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:32.073869+0000 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:32.073869+0000 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:32.075 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.073+0000 7f06b87bd640 1 Processor -- start 2024-03-25T01:59:32.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.073+0000 7f06b87bd640 1 -- start start 2024-03-25T01:59:32.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.073+0000 7f06b87bd640 10 monclient: init 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00a9b20 0x7f06b0146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient(hunting): picked mon.noname-b con 0x7f06b00a9b20 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06b00b3330 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06b0140ac0 0x7f06b014d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient(hunting): picked mon.noname-a con 0x7f06b0140ac0 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b00abb00 con 0x7f06b00b3330 2024-03-25T01:59:32.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f06b00abde0 con 0x7f06b0140ac0 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f06b00b2e80 con 0x7f06b00a9b20 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.074+0000 7f06b87bd640 10 monclient(hunting): authenticate will time out at 1294.357228s 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6559640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00a9b20 0x7f06b0146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6d5a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f06b0140ac0 0x7f06b014d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6559640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00a9b20 0x7f06b0146ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.153:60844/0 (socket says 172.21.15.153:60844) 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6559640 1 -- 172.21.15.153:0/2144094741 learned_addr learned my addr 172.21.15.153:0/2144094741 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6559640 10 monclient(hunting): get_auth_request con 0x7f06b00a9b20 auth_method 0 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6559640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6559640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.075+0000 7f06b6559640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.076+0000 7f06b6559640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.076+0000 7f06b6559640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.076+0000 7f06b6559640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.076+0000 7f06b6d5a640 10 monclient(hunting): get_auth_request con 0x7f06b0140ac0 auth_method 0 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.076+0000 7f06b6d5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.076+0000 7f06b6d5a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.076+0000 7f06b6d5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.077+0000 7f06b6d5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.077+0000 7f06b6d5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.077+0000 7f06b6d5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 1 -- 172.21.15.153:0/2144094741 >> v2:172.21.15.83:3300/0 conn(0x7f06b0140ac0 msgr2=0x7f06b014d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 1 --2- 172.21.15.153:0/2144094741 >> v2:172.21.15.83:3300/0 conn(0x7f06b0140ac0 0x7f06b014d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 1 -- 172.21.15.153:0/2144094741 >> v1:172.21.15.83:6789/0 conn(0x7f06b00b3330 legacy=0x7f06b01403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 10 monclient: _finish_hunting 0 2024-03-25T01:59:32.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 1 monclient: found mon.noname-b 2024-03-25T01:59:32.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:32.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T01:59:32.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.078+0000 7f06b6559640 1 -- 172.21.15.153:0/2144094741 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b00aaef0 con 0x7f06b00a9b20 2024-03-25T01:59:32.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.079+0000 7f06b6559640 1 --2- 172.21.15.153:0/2144094741 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00a9b20 0x7f06b0146ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009870 tx=0x7f06a002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88ebddcec6b1212e server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 1 -- 172.21.15.153:0/2144094741 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a00317c0 con 0x7f06b00a9b20 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: dump: 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: _finish_auth 0 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: _check_auth_tickets 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 1 -- 172.21.15.153:0/2144094741 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a0037070 con 0x7f06b00a9b20 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 5 monclient: authenticate success, global_id 4112 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 4 set_mon_vals no callback set 2024-03-25T01:59:32.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 1 -- 172.21.15.153:0/2144094741 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0031b80 con 0x7f06b00a9b20 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b5557640 10 monclient: dump: 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.088 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 10 monclient: shutdown 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 1 -- 172.21.15.153:0/2144094741 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00a9b20 msgr2=0x7f06b0146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 1 --2- 172.21.15.153:0/2144094741 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00a9b20 0x7f06b0146ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009870 tx=0x7f06a002b860 comp rx=0 tx=0).stop 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 1 -- 172.21.15.153:0/2144094741 shutdown_connections 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 1 --2- 172.21.15.153:0/2144094741 >> v2:172.21.15.83:3300/0 conn(0x7f06b0140ac0 0x7f06b014d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 1 --2- 172.21.15.153:0/2144094741 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00a9b20 0x7f06b0146ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.084+0000 7f06b87bd640 1 -- 172.21.15.153:0/2144094741 >> 172.21.15.153:0/2144094741 conn(0x7f06b00b89b0 msgr2=0x7f06b00b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.085+0000 7f06b87bd640 1 -- 172.21.15.153:0/2144094741 shutdown_connections 2024-03-25T01:59:32.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.085+0000 7f06b87bd640 1 -- 172.21.15.153:0/2144094741 wait complete. 2024-03-25T01:59:32.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.085+0000 7f06b87bd640 10 monclient: build_initial_monmap 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.085+0000 7f06b87bd640 10 monclient: monmap: 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.085+0000 7f06b87bd640 1 librados: starting msgr at 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.085+0000 7f06b87bd640 1 librados: starting objecter 2024-03-25T01:59:32.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.085+0000 7f06b87bd640 1 Processor -- start 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 -- start start 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 librados: setting wanted keys 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 librados: calling monclient init 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient: init 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b00a9b20 0x7f06b013b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): picked mon.noname-a con 0x7f06b00a9b20 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00b3330 0x7f06b01dbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): picked mon.noname-c con 0x7f06b00b3330 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b0140ac0 0x7f06b01cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): picked mon.noname-b con 0x7f06b0140ac0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:32.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f06b0147220 con 0x7f06b00a9b20 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06b00aa150 con 0x7f06b0140ac0 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f06b01473a0 con 0x7f06b00b3330 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.086+0000 7f06b87bd640 10 monclient(hunting): authenticate will time out at 1294.369273s 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6d5a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b0140ac0 0x7f06b01cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6559640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b00a9b20 0x7f06b013b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b5d58640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00b3330 0x7f06b01dbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b5d58640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00b3330 0x7f06b01dbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.153:60854/0 (socket says 172.21.15.153:60854) 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b5d58640 1 -- 172.21.15.153:0/2745177269 learned_addr learned my addr 172.21.15.153:0/2745177269 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:32.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b5d58640 10 monclient(hunting): get_auth_request con 0x7f06b00b3330 auth_method 0 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b5d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b5d58640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b5d58640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6d5a640 10 monclient(hunting): get_auth_request con 0x7f06b0140ac0 auth_method 0 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6d5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6d5a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6d5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6559640 10 monclient(hunting): get_auth_request con 0x7f06b00a9b20 auth_method 0 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6559640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6559640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.087+0000 7f06b6559640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b5d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b5d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b5d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b6d5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b6d5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b6d5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b6559640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b6559640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.088+0000 7f06b6559640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 1 -- 172.21.15.153:0/2745177269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b0140ac0 msgr2=0x7f06b01cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 1 --2- 172.21.15.153:0/2745177269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b0140ac0 0x7f06b01cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 1 -- 172.21.15.153:0/2745177269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b00a9b20 msgr2=0x7f06b013b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 1 --2- 172.21.15.153:0/2745177269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b00a9b20 0x7f06b013b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 10 monclient: _finish_hunting 0 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 1 monclient: found mon.noname-c 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:32.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T01:59:32.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.090+0000 7f06b5d58640 1 -- 172.21.15.153:0/2745177269 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b01cc910 con 0x7f06b00b3330 2024-03-25T01:59:32.098 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:32.094+0000 7f85a6012040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.092+0000 7f06b5d58640 1 --2- 172.21.15.153:0/2745177269 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00b3330 0x7f06b01dbd30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00c400 tx=0x7f06ac00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.092+0000 7f06a6ffd640 10 client.?.objecter ms_handle_connect 0x7f06b00b3330 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.092+0000 7f06a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 1 -- 172.21.15.153:0/2745177269 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac017070 con 0x7f06b00b3330 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: dump: 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.100 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: _finish_auth 0 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 1 -- 172.21.15.153:0/2745177269 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ac0052d0 con 0x7f06b00b3330 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 5 monclient: authenticate success, global_id 4115 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 10 monclient: _renew_subs 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b00ace70 con 0x7f06b00b3330 2024-03-25T01:59:32.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 1 -- 172.21.15.153:0/2745177269 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac003830 con 0x7f06b00b3330 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06a6ffd640 10 monclient: dump: 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.102 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b4d56640 4 set_mon_vals no callback set 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 10 monclient: _renew_subs 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b01dc620 con 0x7f06b00b3330 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.096+0000 7f06b87bd640 1 librados: init done 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.097+0000 7f06b87bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.097+0000 7f06b87bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.097+0000 7f06b87bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:32.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.097+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b004f940 con 0x7f06b00b3330 2024-03-25T01:59:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.100+0000 7f06a6ffd640 1 -- 172.21.15.153:0/2745177269 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f06ac01a070 con 0x7f06b00b3330 2024-03-25T01:59:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.100+0000 7f06a6ffd640 1 -- 172.21.15.153:0/2745177269 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f06ac01b3b0 con 0x7f06b00b3330 2024-03-25T01:59:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.100+0000 7f06a6ffd640 10 client.4115.objecter ms_dispatch 0x7f06b0000b90 osd_map(5..5 src has 1..5) 2024-03-25T01:59:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.100+0000 7f06a6ffd640 3 client.4115.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T01:59:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.100+0000 7f06a6ffd640 3 client.4115.objecter handle_osd_map decoding full epoch 5 2024-03-25T01:59:32.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.100+0000 7f06a6ffd640 20 client.4115.objecter dump_active .. 0 homeless 2024-03-25T01:59:32.131 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:32.130+0000 7f6ff8c3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T01:59:32.212 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:32.211+0000 7f85a6012040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T01:59:32.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.272+0000 7f06a6ffd640 1 -- 172.21.15.153:0/2745177269 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f06ac021020 con 0x7f06b00b3330 2024-03-25T01:59:32.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.272+0000 7f06a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.272+0000 7f06a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:32.288 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:32.287+0000 7f6ff8c3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T01:59:32.316 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:32.315+0000 7f85a6012040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T01:59:32.335 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:32.334+0000 7f6ff8c3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T01:59:32.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.347+0000 7f06b87bd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "415b88b9-7d5a-4162-a686-902197aca9fb", "id": 3}] 2024-03-25T01:59:32.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.347+0000 7f06b87bd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "415b88b9-7d5a-4162-a686-902197aca9fb", "id": 3}] 2024-03-25T01:59:32.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.347+0000 7f06b87bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:32.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.347+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd new", "uuid": "415b88b9-7d5a-4162-a686-902197aca9fb", "id": 3} v 0) -- 0x7f06b00b2490 con 0x7f06b00b3330 2024-03-25T01:59:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.453+0000 7f06a6ffd640 1 -- 172.21.15.153:0/2745177269 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "415b88b9-7d5a-4162-a686-902197aca9fb", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f06ac0115f0 con 0x7f06b00b3330 2024-03-25T01:59:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.453+0000 7f06a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "415b88b9-7d5a-4162-a686-902197aca9fb", "id": 3}] 2024-03-25T01:59:32.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.453+0000 7f06a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:32.455 INFO:teuthology.orchestra.run.smithi153.stdout:3 2024-03-25T01:59:32.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 10 librados: watch_flush enter 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 10 librados: watch_flush exit 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 20 client.4115.objecter shutdown clearing up homeless session... 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 10 client.4115.objecter successfully canceled tick 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 10 monclient: shutdown 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00b3330 msgr2=0x7f06b01dbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 1 --2- 172.21.15.153:0/2745177269 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00b3330 0x7f06b01dbd30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00c400 tx=0x7f06ac00c8c0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 shutdown_connections 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 1 --2- 172.21.15.153:0/2745177269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b0140ac0 0x7f06b01cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 1 --2- 172.21.15.153:0/2745177269 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f06b00b3330 0x7f06b01dbd30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 1 --2- 172.21.15.153:0/2745177269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b00a9b20 0x7f06b013b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.454+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 >> 172.21.15.153:0/2745177269 conn(0x7f06b00b89b0 msgr2=0x7f06b00b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:32.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.455+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 shutdown_connections 2024-03-25T01:59:32.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.455+0000 7f06b87bd640 1 -- 172.21.15.153:0/2745177269 wait complete. 2024-03-25T01:59:32.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.455+0000 7f06b87bd640 1 librados: shutdown 2024-03-25T01:59:32.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 675b8bea-3596-4192-9556-b246aacefe85 4 2024-03-25T01:59:32.491 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:32.489+0000 7f85a6012040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T01:59:32.501 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:32.500+0000 7f6ff8c3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T01:59:32.533 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:32.532+0000 7f85a6012040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T01:59:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.566+0000 7f2a1628f640 10 monclient: get_monmap_and_config 2024-03-25T01:59:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.566+0000 7f2a1628f640 10 monclient: build_initial_monmap 2024-03-25T01:59:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.566+0000 7f2a1628f640 10 monclient: monmap: 2024-03-25T01:59:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:32.568 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:32.567002+0000 2024-03-25T01:59:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:32.567002+0000 2024-03-25T01:59:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:32.569 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:32.570 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:32.570 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:32.570 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:32.570 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.566+0000 7f2a1628f640 1 Processor -- start 2024-03-25T01:59:32.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.566+0000 7f2a1628f640 1 -- start start 2024-03-25T01:59:32.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.566+0000 7f2a1628f640 10 monclient: init 2024-03-25T01:59:32.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:32.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-03-25T01:59:32.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2a100a9b20 0x7f2a10146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient(hunting): picked mon.noname-c con 0x7f2a100a9b20 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:32.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a100b3330 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:32.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a10140ac0 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:32.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a100abb00 con 0x7f2a10140ac0 2024-03-25T01:59:32.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a100abde0 con 0x7f2a100b3330 2024-03-25T01:59:32.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f2a100b2e80 con 0x7f2a100a9b20 2024-03-25T01:59:32.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:32.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.567+0000 7f2a1628f640 10 monclient(hunting): authenticate will time out at 1294.850234s 2024-03-25T01:59:32.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.568+0000 7f2a0ffff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2a100a9b20 0x7f2a10146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.568+0000 7f2a1482c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f2a10140ac0 0x7f2a1014d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.153:46542/0 (socket says 172.21.15.153:46542) 2024-03-25T01:59:32.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.568+0000 7f2a1482c640 1 -- 172.21.15.153:0/4223761141 learned_addr learned my addr 172.21.15.153:0/4223761141 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T01:59:32.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.568+0000 7f2a0ffff640 10 monclient(hunting): get_auth_request con 0x7f2a100a9b20 auth_method 0 2024-03-25T01:59:32.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.568+0000 7f2a0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.568+0000 7f2a0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.568+0000 7f2a0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.569+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.569+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.569+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-03-25T01:59:32.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 1 -- 172.21.15.153:0/4223761141 >> v1:172.21.15.153:6789/0 conn(0x7f2a100b3330 legacy=0x7f2a101403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:32.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 1 -- 172.21.15.153:0/4223761141 >> v1:172.21.15.83:6789/0 conn(0x7f2a10140ac0 legacy=0x7f2a1014d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 10 monclient: _finish_hunting 0 2024-03-25T01:59:32.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 1 monclient: found mon.noname-c 2024-03-25T01:59:32.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:32.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T01:59:32.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.570+0000 7f2a0ffff640 1 -- 172.21.15.153:0/4223761141 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a100aaef0 con 0x7f2a100a9b20 2024-03-25T01:59:32.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.572+0000 7f2a0ffff640 1 --2- 172.21.15.153:0/4223761141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a100a9b20 0x7f2a10146ce0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e980e0fe7280ab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:32.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 1 -- 172.21.15.153:0/4223761141 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00037070 con 0x7f2a100a9b20 2024-03-25T01:59:32.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:32.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: dump: 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.591 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: _finish_auth 0 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: _check_auth_tickets 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 1 -- 172.21.15.153:0/4223761141 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a00030710 con 0x7f2a100a9b20 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a1628f640 5 monclient: authenticate success, global_id 4138 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a1628f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a1628f640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 1 -- 172.21.15.153:0/4223761141 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00030ae0 con 0x7f2a100a9b20 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a1628f640 4 set_mon_vals no callback set 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.577+0000 7f2a0effd640 10 monclient: dump: 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.592 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 10 monclient: shutdown 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 1 -- 172.21.15.153:0/4223761141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a100a9b20 msgr2=0x7f2a10146ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 1 --2- 172.21.15.153:0/4223761141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a100a9b20 0x7f2a10146ce0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a0002b5e0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 1 -- 172.21.15.153:0/4223761141 shutdown_connections 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 1 --2- 172.21.15.153:0/4223761141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a100a9b20 0x7f2a10146ce0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 1 -- 172.21.15.153:0/4223761141 >> 172.21.15.153:0/4223761141 conn(0x7f2a100b89b0 msgr2=0x7f2a100b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 1 -- 172.21.15.153:0/4223761141 shutdown_connections 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 1 -- 172.21.15.153:0/4223761141 wait complete. 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 10 monclient: build_initial_monmap 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.578+0000 7f2a1628f640 10 monclient: monmap: 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:32.594 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.579+0000 7f2a1628f640 1 librados: starting msgr at 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.579+0000 7f2a1628f640 1 librados: starting objecter 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.579+0000 7f2a1628f640 1 Processor -- start 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.579+0000 7f2a1628f640 1 -- start start 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.579+0000 7f2a1628f640 1 librados: setting wanted keys 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.579+0000 7f2a1628f640 1 librados: calling monclient init 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.579+0000 7f2a1628f640 10 monclient: init 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100a9b20 0x7f2a10143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): picked mon.noname-a con 0x7f2a100a9b20 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2a100b3330 0x7f2a101d8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): picked mon.noname-c con 0x7f2a100b3330 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10140ac0 0x7f2a101d8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): picked mon.noname-b con 0x7f2a10140ac0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2a10147220 con 0x7f2a100a9b20 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a100aa1d0 con 0x7f2a10140ac0 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2a101473a0 con 0x7f2a100b3330 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1628f640 10 monclient(hunting): authenticate will time out at 1294.862831s 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a0ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100a9b20 0x7f2a10143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a1482c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10140ac0 0x7f2a101d8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.580+0000 7f2a0f7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2a100b3330 0x7f2a101d8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100a9b20 0x7f2a10143900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.153:37768/0 (socket says 172.21.15.153:37768) 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 1 -- 172.21.15.153:0/1616845391 learned_addr learned my addr 172.21.15.153:0/1616845391 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:32.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 10 monclient(hunting): get_auth_request con 0x7f2a100a9b20 auth_method 0 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a1482c640 10 monclient(hunting): get_auth_request con 0x7f2a10140ac0 auth_method 0 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a1482c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a1482c640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a1482c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0f7fe640 10 monclient(hunting): get_auth_request con 0x7f2a100b3330 auth_method 0 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a1482c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a1482c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a1482c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.581+0000 7f2a0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 10 monclient(hunting): handle_auth_done global_id 4140 payload 824 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 1 -- 172.21.15.153:0/1616845391 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2a100b3330 msgr2=0x7f2a101d8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2a100b3330 0x7f2a101d8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 1 -- 172.21.15.153:0/1616845391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10140ac0 msgr2=0x7f2a101d8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10140ac0 0x7f2a101d8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 10 monclient: _finish_hunting 0 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 1 monclient: found mon.noname-a 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a0ffff640 1 -- 172.21.15.153:0/1616845391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a101429e0 con 0x7f2a100a9b20 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.584+0000 7f2a1482c640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10140ac0 0x7f2a101d8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.586+0000 7f2a0ffff640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100a9b20 0x7f2a10143900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.586+0000 7f2a0cff9640 10 client.?.objecter ms_handle_connect 0x7f2a100a9b20 2024-03-25T01:59:32.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.586+0000 7f2a0cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:32.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 1 -- 172.21.15.153:0/1616845391 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00042070 con 0x7f2a100a9b20 2024-03-25T01:59:32.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: dump: 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: _finish_auth 0 2024-03-25T01:59:32.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: _check_auth_tickets 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 1 -- 172.21.15.153:0/1616845391 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a00032a40 con 0x7f2a100a9b20 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 5 monclient: authenticate success, global_id 4140 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 10 monclient: _renew_subs 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a101d9410 con 0x7f2a100a9b20 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 1 -- 172.21.15.153:0/1616845391 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00037040 con 0x7f2a100a9b20 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0cff9640 10 monclient: dump: 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.603 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a0e7fc640 4 set_mon_vals no callback set 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 10 monclient: _renew_subs 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a10143250 con 0x7f2a100a9b20 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.590+0000 7f2a1628f640 1 librados: init done 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.591+0000 7f2a1628f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.591+0000 7f2a1628f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.591+0000 7f2a1628f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.591+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a101cec30 con 0x7f2a100a9b20 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.593+0000 7f2a0cff9640 1 -- 172.21.15.153:0/1616845391 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a00045070 con 0x7f2a100a9b20 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.593+0000 7f2a0cff9640 1 -- 172.21.15.153:0/1616845391 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f2a00051bb0 con 0x7f2a100a9b20 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.593+0000 7f2a0cff9640 10 client.4140.objecter ms_dispatch 0x7f2a10000b90 osd_map(6..6 src has 1..6) 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.593+0000 7f2a0cff9640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.593+0000 7f2a0cff9640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2024-03-25T01:59:32.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.593+0000 7f2a0cff9640 20 client.4140.objecter dump_active .. 0 homeless 2024-03-25T01:59:32.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.689+0000 7f2a0cff9640 1 -- 172.21.15.153:0/1616845391 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2a00051e00 con 0x7f2a100a9b20 2024-03-25T01:59:32.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.689+0000 7f2a0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.689+0000 7f2a0cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:32.696 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:32.694+0000 7f85a6012040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T01:59:32.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.762+0000 7f2a1628f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "675b8bea-3596-4192-9556-b246aacefe85", "id": 4}] 2024-03-25T01:59:32.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.762+0000 7f2a1628f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "675b8bea-3596-4192-9556-b246aacefe85", "id": 4}] 2024-03-25T01:59:32.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.762+0000 7f2a1628f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:32.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.762+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "675b8bea-3596-4192-9556-b246aacefe85", "id": 4} v 0) -- 0x7f2a10143480 con 0x7f2a100a9b20 2024-03-25T01:59:32.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.791+0000 7f2a0cff9640 1 -- 172.21.15.153:0/1616845391 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "675b8bea-3596-4192-9556-b246aacefe85", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f2a00051480 con 0x7f2a100a9b20 2024-03-25T01:59:32.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.791+0000 7f2a0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "675b8bea-3596-4192-9556-b246aacefe85", "id": 4}] 2024-03-25T01:59:32.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.791+0000 7f2a0cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:32.793 INFO:teuthology.orchestra.run.smithi153.stdout:4 2024-03-25T01:59:32.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 10 librados: watch_flush enter 2024-03-25T01:59:32.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 10 librados: watch_flush exit 2024-03-25T01:59:32.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 20 client.4140.objecter shutdown clearing up homeless session... 2024-03-25T01:59:32.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 10 client.4140.objecter successfully canceled tick 2024-03-25T01:59:32.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 10 monclient: shutdown 2024-03-25T01:59:32.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:32.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100a9b20 msgr2=0x7f2a10143900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100a9b20 0x7f2a10143900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2a000097a0 tx=0x7f2a000046c0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 shutdown_connections 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a10140ac0 0x7f2a101d8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2a100b3330 0x7f2a101d8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 --2- 172.21.15.153:0/1616845391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2a100a9b20 0x7f2a10143900 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 >> 172.21.15.153:0/1616845391 conn(0x7f2a100b89b0 msgr2=0x7f2a100b80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 shutdown_connections 2024-03-25T01:59:32.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 -- 172.21.15.153:0/1616845391 wait complete. 2024-03-25T01:59:32.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.792+0000 7f2a1628f640 1 librados: shutdown 2024-03-25T01:59:32.804 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 74d4302c-0ae2-4627-a3a9-f4e37ad1ae50 5 2024-03-25T01:59:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.944+0000 7f56fc58f640 10 monclient: get_monmap_and_config 2024-03-25T01:59:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.944+0000 7f56fc58f640 10 monclient: build_initial_monmap 2024-03-25T01:59:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.944+0000 7f56fc58f640 10 monclient: monmap: 2024-03-25T01:59:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:32.946 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:32.945611+0000 2024-03-25T01:59:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:32.945611+0000 2024-03-25T01:59:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:32.947 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:32.948 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:32.948 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:32.948 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:32.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.945+0000 7f56fc58f640 1 Processor -- start 2024-03-25T01:59:32.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.945+0000 7f56fc58f640 1 -- start start 2024-03-25T01:59:32.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.945+0000 7f56fc58f640 10 monclient: init 2024-03-25T01:59:32.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:32.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-03-25T01:59:32.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient(hunting): picked mon.noname-b con 0x7f56f406b870 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:32.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56f413e360 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:32.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56f4144cc0 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:32.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f40a9c40 con 0x7f56f413e360 2024-03-25T01:59:32.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f40a9f20 con 0x7f56f4144cc0 2024-03-25T01:59:32.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f56f406bc50 con 0x7f56f406b870 2024-03-25T01:59:32.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:32.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fc58f640 10 monclient(hunting): authenticate will time out at 1295.228987s 2024-03-25T01:59:32.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fab2c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f56f4144cc0 0x7f56f414b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:57946/0 (socket says 172.21.15.153:57946) 2024-03-25T01:59:32.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.946+0000 7f56fab2c640 1 -- 172.21.15.153:0/2903628944 learned_addr learned my addr 172.21.15.153:0/2903628944 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T01:59:32.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.947+0000 7f56fa32b640 1 --2- 172.21.15.153:0/2903628944 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.947+0000 7f56fa32b640 10 monclient(hunting): get_auth_request con 0x7f56f406b870 auth_method 0 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.947+0000 7f56fa32b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.947+0000 7f56fa32b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.947+0000 7f56fa32b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.948+0000 7f56fa32b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.948+0000 7f56fa32b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.948+0000 7f56fa32b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 1 -- 172.21.15.153:0/2903628944 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2206210492 0 0) 0x7f56f40a9f20 con 0x7f56f4144cc0 2024-03-25T01:59:32.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): _negotiate 2024-03-25T01:59:32.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): my global_id is 4144 2024-03-25T01:59:32.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 1 -- 172.21.15.153:0/2903628944 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e0003520 con 0x7f56f4144cc0 2024-03-25T01:59:32.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 1 -- 172.21.15.153:0/2903628944 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850243721 0 0) 0x7f56f40a9c40 con 0x7f56f413e360 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): _negotiate 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 10 monclient(hunting): my global_id is 4143 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.949+0000 7f56f9329640 1 -- 172.21.15.153:0/2903628944 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f40a9f20 con 0x7f56f413e360 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 1 -- 172.21.15.153:0/2903628944 >> v1:172.21.15.153:6789/0 conn(0x7f56f4144cc0 legacy=0x7f56f414b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 1 -- 172.21.15.153:0/2903628944 >> v1:172.21.15.83:6789/0 conn(0x7f56f413e360 legacy=0x7f56f41445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 10 monclient: _finish_hunting 0 2024-03-25T01:59:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 1 monclient: found mon.noname-b 2024-03-25T01:59:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T01:59:32.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.950+0000 7f56fa32b640 1 -- 172.21.15.153:0/2903628944 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f40b0990 con 0x7f56f406b870 2024-03-25T01:59:32.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.953+0000 7f56fa32b640 1 --2- 172.21.15.153:0/2903628944 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413de20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f56f40b0bc0 tx=0x7f56e402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46fc5456a419f3ea server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:32.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 1 -- 172.21.15.153:0/2903628944 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4035020 con 0x7f56f406b870 2024-03-25T01:59:32.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: dump: 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: _finish_auth 0 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: _check_auth_tickets 2024-03-25T01:59:32.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 1 -- 172.21.15.153:0/2903628944 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e4031890 con 0x7f56f406b870 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 1 -- 172.21.15.153:0/2903628944 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4031c60 con 0x7f56f406b870 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56fc58f640 5 monclient: authenticate success, global_id 4121 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56fc58f640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56fc58f640 4 set_mon_vals no callback set 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.957+0000 7f56f9329640 10 monclient: dump: 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.967 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 10 monclient: shutdown 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 1 -- 172.21.15.153:0/2903628944 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 msgr2=0x7f56f413de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 1 --2- 172.21.15.153:0/2903628944 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413de20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f56f40b0bc0 tx=0x7f56e402b930 comp rx=0 tx=0).stop 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 1 -- 172.21.15.153:0/2903628944 shutdown_connections 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 1 --2- 172.21.15.153:0/2903628944 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413de20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 1 -- 172.21.15.153:0/2903628944 >> 172.21.15.153:0/2903628944 conn(0x7f56f40b5b50 msgr2=0x7f56f40b5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 1 -- 172.21.15.153:0/2903628944 shutdown_connections 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 1 -- 172.21.15.153:0/2903628944 wait complete. 2024-03-25T01:59:32.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 10 monclient: build_initial_monmap 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.958+0000 7f56fc58f640 10 monclient: monmap: 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 1 librados: starting msgr at 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 1 librados: starting objecter 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 1 Processor -- start 2024-03-25T01:59:32.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 1 -- start start 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 1 librados: setting wanted keys 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 1 librados: calling monclient init 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 10 monclient: init 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.959+0000 7f56fc58f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): picked mon.noname-c con 0x7f56f406b870 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f413e360 0x7f56f4070780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): picked mon.noname-a con 0x7f56f413e360 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f4144cc0 0x7f56f41407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): picked mon.noname-b con 0x7f56f4144cc0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56f40b12e0 con 0x7f56f413e360 2024-03-25T01:59:32.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f56f413e7a0 con 0x7f56f4144cc0 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40b1460 con 0x7f56f406b870 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fc58f640 10 monclient(hunting): authenticate will time out at 1295.242782s 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fab2c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f4144cc0 0x7f56f41407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56f9b2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f413e360 0x7f56f4070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fa32b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fab2c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f4144cc0 0x7f56f41407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33612/0 (socket says 172.21.15.153:33612) 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fab2c640 1 -- 172.21.15.153:0/2653266622 learned_addr learned my addr 172.21.15.153:0/2653266622 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fab2c640 10 monclient(hunting): get_auth_request con 0x7f56f4144cc0 auth_method 0 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fab2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fab2c640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.960+0000 7f56fab2c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56f9b2a640 10 monclient(hunting): get_auth_request con 0x7f56f413e360 auth_method 0 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56f9b2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56f9b2a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56f9b2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fa32b640 10 monclient(hunting): get_auth_request con 0x7f56f406b870 auth_method 0 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fa32b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fa32b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fa32b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fab2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fab2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fab2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56f9b2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56f9b2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56f9b2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fa32b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fa32b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.961+0000 7f56fa32b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.962+0000 7f56fa32b640 1 -- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 msgr2=0x7f56f413fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.962+0000 7f56fa32b640 1 -- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 msgr2=0x7f56f413fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.962+0000 7f56fa32b640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.962+0000 7f56fa32b640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413fc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 1 -- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 msgr2=0x7f56f413fc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413fc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 1 -- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f413e360 msgr2=0x7f56f4070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f413e360 0x7f56f4070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 10 monclient: _finish_hunting 0 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 1 monclient: found mon.noname-b 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:32.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T01:59:32.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56fab2c640 1 -- 172.21.15.153:0/2653266622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f41417c0 con 0x7f56f4144cc0 2024-03-25T01:59:32.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.964+0000 7f56f9b2a640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f413e360 0x7f56f4070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.966+0000 7f56fab2c640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f4144cc0 0x7f56f41407e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00cb70 tx=0x7f56ec00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.966+0000 7f56eaffd640 10 client.?.objecter ms_handle_connect 0x7f56f4144cc0 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.966+0000 7f56eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56eaffd640 1 -- 172.21.15.153:0/2653266622 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec017070 con 0x7f56f4144cc0 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56eaffd640 10 monclient: dump: 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:32.975 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56eaffd640 10 monclient: _finish_auth 0 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56eaffd640 10 monclient: _check_auth_tickets 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 5 monclient: authenticate success, global_id 4147 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 10 monclient: _renew_subs 2024-03-25T01:59:32.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f4141a30 con 0x7f56f4144cc0 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 10 monclient: _renew_subs 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f4143ba0 con 0x7f56f4144cc0 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.968+0000 7f56fc58f640 1 librados: init done 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.969+0000 7f56fc58f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.969+0000 7f56fc58f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.969+0000 7f56fc58f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:32.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:32.969+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f404f940 con 0x7f56f4144cc0 2024-03-25T01:59:33.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 1 -- 172.21.15.153:0/2653266622 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec00ad40 con 0x7f56f4144cc0 2024-03-25T01:59:33.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 1 -- 172.21.15.153:0/2653266622 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec021060 con 0x7f56f4144cc0 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 10 monclient: dump: 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.073 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 1 -- 172.21.15.153:0/2653266622 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f56ec01a070 con 0x7f56f4144cc0 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56f8b28640 4 set_mon_vals no callback set 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 1 -- 172.21.15.153:0/2653266622 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f56ec012cf0 con 0x7f56f4144cc0 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 10 client.4147.objecter ms_dispatch 0x7f56f4000b90 osd_map(7..7 src has 1..7) 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 1 -- 172.21.15.153:0/2653266622 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f56ec0070b0 con 0x7f56f4144cc0 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:33.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.071+0000 7f56eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:33.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.144+0000 7f56fc58f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "74d4302c-0ae2-4627-a3a9-f4e37ad1ae50", "id": 5}] 2024-03-25T01:59:33.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.144+0000 7f56fc58f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "74d4302c-0ae2-4627-a3a9-f4e37ad1ae50", "id": 5}] 2024-03-25T01:59:33.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.144+0000 7f56fc58f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:33.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.144+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "74d4302c-0ae2-4627-a3a9-f4e37ad1ae50", "id": 5} v 0) -- 0x7f56f4140430 con 0x7f56f4144cc0 2024-03-25T01:59:33.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.256+0000 7f56eaffd640 1 -- 172.21.15.153:0/2653266622 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "74d4302c-0ae2-4627-a3a9-f4e37ad1ae50", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f56ec012770 con 0x7f56f4144cc0 2024-03-25T01:59:33.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.256+0000 7f56eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "74d4302c-0ae2-4627-a3a9-f4e37ad1ae50", "id": 5}] 2024-03-25T01:59:33.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.256+0000 7f56eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:33.258 INFO:teuthology.orchestra.run.smithi153.stdout:5 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 10 librados: watch_flush enter 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 10 librados: watch_flush exit 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 10 client.4147.objecter successfully canceled tick 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 10 monclient: shutdown 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f4144cc0 msgr2=0x7f56f41407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f4144cc0 0x7f56f41407e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00cb70 tx=0x7f56ec00a430 comp rx=0 tx=0).stop 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 shutdown_connections 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56f4144cc0 0x7f56f41407e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56f413e360 0x7f56f4070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 --2- 172.21.15.153:0/2653266622 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f56f406b870 0x7f56f413fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 >> 172.21.15.153:0/2653266622 conn(0x7f56f40b5b50 msgr2=0x7f56f40b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 shutdown_connections 2024-03-25T01:59:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 -- 172.21.15.153:0/2653266622 wait complete. 2024-03-25T01:59:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.257+0000 7f56fc58f640 1 librados: shutdown 2024-03-25T01:59:33.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new b754b718-408c-4a47-8d2c-a66d50ed18e5 6 2024-03-25T01:59:33.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.370+0000 7f28e1afd640 10 monclient: get_monmap_and_config 2024-03-25T01:59:33.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.370+0000 7f28e1afd640 10 monclient: build_initial_monmap 2024-03-25T01:59:33.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.370+0000 7f28e1afd640 10 monclient: monmap: 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:33.370925+0000 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:33.370925+0000 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:33.372 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 1 Processor -- start 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 1 -- start start 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient: init 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28dc0b1680 0x7f28dc06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient(hunting): picked mon.noname-a con 0x7f28dc0b1680 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:33.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28dc0afda0 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28dc0adc90 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28dc0aed40 con 0x7f28dc0adc90 2024-03-25T01:59:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28dc0af020 con 0x7f28dc0afda0 2024-03-25T01:59:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f28dc073090 con 0x7f28dc0b1680 2024-03-25T01:59:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.371+0000 7f28e1afd640 10 monclient(hunting): authenticate will time out at 1295.654566s 2024-03-25T01:59:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.372+0000 7f28db7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f28dc0b1680 0x7f28dc06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.372+0000 7f28dbfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f28dc0adc90 0x7f28dc14d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.153:46566/0 (socket says 172.21.15.153:46566) 2024-03-25T01:59:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.372+0000 7f28dbfff640 1 -- 172.21.15.153:0/3061272911 learned_addr learned my addr 172.21.15.153:0/3061272911 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-03-25T01:59:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.373+0000 7f28db7fe640 10 monclient(hunting): get_auth_request con 0x7f28dc0b1680 auth_method 0 2024-03-25T01:59:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.373+0000 7f28db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.373+0000 7f28db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.373+0000 7f28db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.373+0000 7f28db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.373+0000 7f28db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.373+0000 7f28db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 10 monclient(hunting): handle_auth_done global_id 4149 payload 274 2024-03-25T01:59:33.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 1 -- 172.21.15.153:0/3061272911 >> v1:172.21.15.153:6789/0 conn(0x7f28dc0afda0 legacy=0x7f28dc146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 1 -- 172.21.15.153:0/3061272911 >> v1:172.21.15.83:6789/0 conn(0x7f28dc0adc90 legacy=0x7f28dc14d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 10 monclient: _finish_hunting 0 2024-03-25T01:59:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 1 monclient: found mon.noname-a 2024-03-25T01:59:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:33.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.375+0000 7f28db7fe640 1 -- 172.21.15.153:0/3061272911 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28dc072db0 con 0x7f28dc0b1680 2024-03-25T01:59:33.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.380+0000 7f28db7fe640 1 --2- 172.21.15.153:0/3061272911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0b1680 0x7f28dc06b600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0097a0 tx=0x7f28cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=757f73a72f07e5b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 1 -- 172.21.15.153:0/3061272911 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc035020 con 0x7f28dc0b1680 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: dump: 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.385 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: _finish_auth 0 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: _check_auth_tickets 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 1 -- 172.21.15.153:0/3061272911 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28cc031830 con 0x7f28dc0b1680 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 5 monclient: authenticate success, global_id 4149 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:33.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 4 set_mon_vals no callback set 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 1 -- 172.21.15.153:0/3061272911 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc031c00 con 0x7f28dc0b1680 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28da7fc640 10 monclient: dump: 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.387 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 10 monclient: shutdown 2024-03-25T01:59:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 1 -- 172.21.15.153:0/3061272911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0b1680 msgr2=0x7f28dc06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.384+0000 7f28e1afd640 1 --2- 172.21.15.153:0/3061272911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0b1680 0x7f28dc06b600 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0097a0 tx=0x7f28cc02b8d0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 1 -- 172.21.15.153:0/3061272911 shutdown_connections 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 1 --2- 172.21.15.153:0/3061272911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0b1680 0x7f28dc06b600 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 1 -- 172.21.15.153:0/3061272911 >> 172.21.15.153:0/3061272911 conn(0x7f28dc06cff0 msgr2=0x7f28dc0b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 1 -- 172.21.15.153:0/3061272911 shutdown_connections 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 1 -- 172.21.15.153:0/3061272911 wait complete. 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 10 monclient: build_initial_monmap 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 10 monclient: monmap: 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:33.393 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 1 librados: starting msgr at 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.385+0000 7f28e1afd640 1 librados: starting objecter 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 Processor -- start 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 -- start start 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 librados: setting wanted keys 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 librados: calling monclient init 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient: init 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:33.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0adc90 0x7f28dc13b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): picked mon.noname-a con 0x7f28dc0adc90 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f28dc0afda0 0x7f28dc1dbf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): picked mon.noname-c con 0x7f28dc0afda0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28dc0b1680 0x7f28dc1dc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): picked mon.noname-b con 0x7f28dc0b1680 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc146ca0 con 0x7f28dc0adc90 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28dc0b1a60 con 0x7f28dc0b1680 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f28dc1dcc50 con 0x7f28dc0afda0 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:33.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.386+0000 7f28e1afd640 10 monclient(hunting): authenticate will time out at 1295.669539s 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28dbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28dc0b1680 0x7f28dc1dc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0adc90 0x7f28dc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28daffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f28dc0afda0 0x7f28dc1dbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28dbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28dc0b1680 0x7f28dc1dc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33624/0 (socket says 172.21.15.153:33624) 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28dbfff640 1 -- 172.21.15.153:0/2931256189 learned_addr learned my addr 172.21.15.153:0/2931256189 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28dbfff640 10 monclient(hunting): get_auth_request con 0x7f28dc0b1680 auth_method 0 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28db7fe640 10 monclient(hunting): get_auth_request con 0x7f28dc0adc90 auth_method 0 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28daffd640 10 monclient(hunting): get_auth_request con 0x7f28dc0afda0 auth_method 0 2024-03-25T01:59:33.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.387+0000 7f28daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.388+0000 7f28daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 10 monclient(hunting): handle_auth_done global_id 4155 payload 824 2024-03-25T01:59:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 1 -- 172.21.15.153:0/2931256189 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f28dc0afda0 msgr2=0x7f28dc1dbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 1 --2- 172.21.15.153:0/2931256189 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f28dc0afda0 0x7f28dc1dbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 1 -- 172.21.15.153:0/2931256189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28dc0b1680 msgr2=0x7f28dc1dc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 1 --2- 172.21.15.153:0/2931256189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28dc0b1680 0x7f28dc1dc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 10 monclient: _finish_hunting 0 2024-03-25T01:59:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 1 monclient: found mon.noname-a 2024-03-25T01:59:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:33.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.390+0000 7f28db7fe640 1 -- 172.21.15.153:0/2931256189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28dc140ea0 con 0x7f28dc0adc90 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.393+0000 7f28db7fe640 1 --2- 172.21.15.153:0/2931256189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0adc90 0x7f28dc13b790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f28cc033000 tx=0x7f28cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.393+0000 7f28c3fff640 10 client.?.objecter ms_handle_connect 0x7f28dc0adc90 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.393+0000 7f28c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28c3fff640 1 -- 172.21.15.153:0/2931256189 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc03aa70 con 0x7f28dc0adc90 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28c3fff640 10 monclient: dump: 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.401 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28c3fff640 10 monclient: _finish_auth 0 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28c3fff640 10 monclient: _check_auth_tickets 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 5 monclient: authenticate success, global_id 4155 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 10 monclient: _renew_subs 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28dc13d130 con 0x7f28dc0adc90 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 10 monclient: _renew_subs 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28dc13d6a0 con 0x7f28dc0adc90 2024-03-25T01:59:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.395+0000 7f28e1afd640 1 librados: init done 2024-03-25T01:59:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.396+0000 7f28e1afd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.396+0000 7f28e1afd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.396+0000 7f28e1afd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.396+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28dc04f940 con 0x7f28dc0adc90 2024-03-25T01:59:33.403 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:33.398+0000 7f6ff8c3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T01:59:33.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28c3fff640 1 -- 172.21.15.153:0/2931256189 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28cc0317d0 con 0x7f28dc0adc90 2024-03-25T01:59:33.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28c3fff640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:33.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28c3fff640 1 -- 172.21.15.153:0/2931256189 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc03ae10 con 0x7f28dc0adc90 2024-03-25T01:59:33.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:33.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28c3fff640 10 monclient: dump: 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28d9ffb640 4 set_mon_vals no callback set 2024-03-25T01:59:33.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.491+0000 7f28c3fff640 1 -- 172.21.15.153:0/2931256189 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f28cc04b060 con 0x7f28dc0adc90 2024-03-25T01:59:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.492+0000 7f28c3fff640 1 -- 172.21.15.153:0/2931256189 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f28cc0399a0 con 0x7f28dc0adc90 2024-03-25T01:59:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.492+0000 7f28c3fff640 10 client.4155.objecter ms_dispatch 0x7f28dc000b90 osd_map(8..8 src has 1..8) 2024-03-25T01:59:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.492+0000 7f28c3fff640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T01:59:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.492+0000 7f28c3fff640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T01:59:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.492+0000 7f28c3fff640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T01:59:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.496+0000 7f28c3fff640 1 -- 172.21.15.153:0/2931256189 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f28cc03f070 con 0x7f28dc0adc90 2024-03-25T01:59:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.496+0000 7f28c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.496+0000 7f28c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:33.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.568+0000 7f28e1afd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b754b718-408c-4a47-8d2c-a66d50ed18e5", "id": 6}] 2024-03-25T01:59:33.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.568+0000 7f28e1afd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b754b718-408c-4a47-8d2c-a66d50ed18e5", "id": 6}] 2024-03-25T01:59:33.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.568+0000 7f28e1afd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:33.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.568+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b754b718-408c-4a47-8d2c-a66d50ed18e5", "id": 6} v 0) -- 0x7f28dc146570 con 0x7f28dc0adc90 2024-03-25T01:59:33.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.616+0000 7f28c3fff640 1 -- 172.21.15.153:0/2931256189 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b754b718-408c-4a47-8d2c-a66d50ed18e5", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f28cc039ba0 con 0x7f28dc0adc90 2024-03-25T01:59:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.616+0000 7f28c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b754b718-408c-4a47-8d2c-a66d50ed18e5", "id": 6}] 2024-03-25T01:59:33.618 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.616+0000 7f28c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:33.618 INFO:teuthology.orchestra.run.smithi153.stdout:6 2024-03-25T01:59:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 10 librados: watch_flush enter 2024-03-25T01:59:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 10 librados: watch_flush exit 2024-03-25T01:59:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T01:59:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 10 client.4155.objecter successfully canceled tick 2024-03-25T01:59:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 10 monclient: shutdown 2024-03-25T01:59:33.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0adc90 msgr2=0x7f28dc13b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 --2- 172.21.15.153:0/2931256189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0adc90 0x7f28dc13b790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f28cc033000 tx=0x7f28cc0046c0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 shutdown_connections 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 --2- 172.21.15.153:0/2931256189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28dc0b1680 0x7f28dc1dc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 --2- 172.21.15.153:0/2931256189 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f28dc0afda0 0x7f28dc1dbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 --2- 172.21.15.153:0/2931256189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28dc0adc90 0x7f28dc13b790 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 >> 172.21.15.153:0/2931256189 conn(0x7f28dc06cff0 msgr2=0x7f28dc13f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 shutdown_connections 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 -- 172.21.15.153:0/2931256189 wait complete. 2024-03-25T01:59:33.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.617+0000 7f28e1afd640 1 librados: shutdown 2024-03-25T01:59:33.630 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd new 23e89d51-cb05-4ca8-a5f0-1bf573aad7be 7 2024-03-25T01:59:33.701 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:33.699+0000 7f6ff8c3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T01:59:33.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.727+0000 7f2580d04640 10 monclient: get_monmap_and_config 2024-03-25T01:59:33.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.727+0000 7f2580d04640 10 monclient: build_initial_monmap 2024-03-25T01:59:33.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.727+0000 7f2580d04640 10 monclient: monmap: 2024-03-25T01:59:33.729 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:33.729 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.729 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:59:33.728799+0000 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:59:33.728799+0000 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.728+0000 7f2580d04640 1 Processor -- start 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.728+0000 7f2580d04640 1 -- start start 2024-03-25T01:59:33.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.728+0000 7f2580d04640 10 monclient: init 2024-03-25T01:59:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-03-25T01:59:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f257c0a9b20 0x7f257c146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient(hunting): picked mon.noname-a con 0x7f257c0a9b20 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f257c0b3330 0x7f257c1403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient(hunting): picked mon.noname-c con 0x7f257c0b3330 addr v2:172.21.15.153:3300/0 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f257c0a9f00 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257c0abb00 con 0x7f257c0a9f00 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f257c0abde0 con 0x7f257c0a9b20 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f257c0b2e80 con 0x7f257c0b3330 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2580d04640 10 monclient(hunting): authenticate will time out at 1296.012146s 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.729+0000 7f2579d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f257c0b3330 0x7f257c1403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f257a59c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f257c0a9b20 0x7f257c146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f257c0b3330 0x7f257c1403b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33638/0 (socket says 172.21.15.153:33638) 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 1 -- 172.21.15.153:0/607884864 learned_addr learned my addr 172.21.15.153:0/607884864 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 10 monclient(hunting): get_auth_request con 0x7f257c0b3330 auth_method 0 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f257a59c640 10 monclient(hunting): get_auth_request con 0x7f257c0a9b20 auth_method 0 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f257a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f257a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f257a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.730+0000 7f2579d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.731+0000 7f257a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.731+0000 7f257a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.731+0000 7f257a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 1 -- 172.21.15.153:0/607884864 >> v2:172.21.15.83:3300/0 conn(0x7f257c0a9b20 msgr2=0x7f257c146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 1 --2- 172.21.15.153:0/607884864 >> v2:172.21.15.83:3300/0 conn(0x7f257c0a9b20 0x7f257c146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 1 -- 172.21.15.153:0/607884864 >> v1:172.21.15.83:6789/0 conn(0x7f257c0a9f00 legacy=0x7f257c143140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 10 monclient: _finish_hunting 0 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 1 monclient: found mon.noname-c 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.732+0000 7f2579d9b640 1 -- 172.21.15.153:0/607884864 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257c0aaef0 con 0x7f257c0b3330 2024-03-25T01:59:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.734+0000 7f2579d9b640 1 --2- 172.21.15.153:0/607884864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1403b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f257c0b6530 tx=0x7f256c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a7c09a6a0cb9c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 1 -- 172.21.15.153:0/607884864 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c02e070 con 0x7f257c0b3330 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: dump: 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: _finish_auth 0 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: _check_auth_tickets 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 1 -- 172.21.15.153:0/607884864 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f256c037070 con 0x7f257c0b3330 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f2580d04640 5 monclient: authenticate success, global_id 4153 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f2580d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f2580d04640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f2580d04640 4 set_mon_vals no callback set 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 1 -- 172.21.15.153:0/607884864 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c031a00 con 0x7f257c0b3330 2024-03-25T01:59:33.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.738+0000 7f257959a640 10 monclient: dump: 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.742 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 10 monclient: shutdown 2024-03-25T01:59:33.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:33.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 -- 172.21.15.153:0/607884864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 msgr2=0x7f257c1403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 --2- 172.21.15.153:0/607884864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1403b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f257c0b6530 tx=0x7f256c02b880 comp rx=0 tx=0).stop 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 -- 172.21.15.153:0/607884864 shutdown_connections 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 --2- 172.21.15.153:0/607884864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1403b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 --2- 172.21.15.153:0/607884864 >> v2:172.21.15.83:3300/0 conn(0x7f257c0a9b20 0x7f257c146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 -- 172.21.15.153:0/607884864 >> 172.21.15.153:0/607884864 conn(0x7f257c0b89b0 msgr2=0x7f257c0b8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 -- 172.21.15.153:0/607884864 shutdown_connections 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.739+0000 7f2580d04640 1 -- 172.21.15.153:0/607884864 wait complete. 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 10 monclient: build_initial_monmap 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 10 monclient: monmap: 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-03-25T01:59:33.747 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 1 librados: starting msgr at 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 1 librados: starting objecter 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 1 Processor -- start 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 1 -- start start 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 1 librados: setting wanted keys 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 1 librados: calling monclient init 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.741+0000 7f2580d04640 10 monclient: init 2024-03-25T01:59:33.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f257c0a9b20 0x7f257c1cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): picked mon.noname-c con 0x7f257c0a9b20 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f257c0a9f00 0x7f257c1cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): picked mon.noname-a con 0x7f257c0a9f00 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): picked mon.noname-b con 0x7f257c0b3330 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f257c1408f0 con 0x7f257c0a9f00 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f257c147470 con 0x7f257c0b3330 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f257c140a70 con 0x7f257c0a9b20 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:33.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.742+0000 7f2580d04640 10 monclient(hunting): authenticate will time out at 1296.025392s 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f2579d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f257c0a9f00 0x7f257c1cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257a59c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f257c0a9b20 0x7f257c1cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33644/0 (socket says 172.21.15.153:33644) 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 1 -- 172.21.15.153:0/3461709967 learned_addr learned my addr 172.21.15.153:0/3461709967 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 10 monclient(hunting): get_auth_request con 0x7f257c0b3330 auth_method 0 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f2579d9b640 10 monclient(hunting): get_auth_request con 0x7f257c0a9f00 auth_method 0 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f2579d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f2579d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f2579d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.743+0000 7f257ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257a59c640 10 monclient(hunting): get_auth_request con 0x7f257c0a9b20 auth_method 0 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f2579d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f2579d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f2579d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:33.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.744+0000 7f257a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:33.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2024-03-25T01:59:33.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 1 -- 172.21.15.153:0/3461709967 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f257c0a9b20 msgr2=0x7f257c1cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f257c0a9b20 0x7f257c1cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 1 -- 172.21.15.153:0/3461709967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f257c0a9f00 msgr2=0x7f257c1cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:33.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f257c0a9f00 0x7f257c1cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 10 monclient: _finish_hunting 0 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 1 monclient: found mon.noname-b 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f257ad9d640 1 -- 172.21.15.153:0/3461709967 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257c1cc840 con 0x7f257c0b3330 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.746+0000 7f2579d9b640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f257c0a9f00 0x7f257c1cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.748+0000 7f257ad9d640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1ceab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f257400c7a0 tx=0x7f257400cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.748+0000 7f256affd640 10 client.?.objecter ms_handle_connect 0x7f257c0b3330 2024-03-25T01:59:33.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.749+0000 7f256affd640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2574003b70 con 0x7f257c0b3330 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: dump: 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.755 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: _finish_auth 0 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: _check_auth_tickets 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2574003d30 con 0x7f257c0b3330 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 5 monclient: authenticate success, global_id 4156 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 10 monclient: _renew_subs 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257400aa90 con 0x7f257c0b3330 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257c0acd00 con 0x7f257c0b3330 2024-03-25T01:59:33.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2578d99640 4 set_mon_vals no callback set 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f256affd640 10 monclient: dump: 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T01:59:33.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 10 monclient: _renew_subs 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f257c1cfc20 con 0x7f257c0b3330 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.753+0000 7f2580d04640 1 librados: init done 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.754+0000 7f2580d04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.754+0000 7f2580d04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.754+0000 7f2580d04640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:33.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.754+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f257c04f940 con 0x7f257c0b3330 2024-03-25T01:59:33.761 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:33.758+0000 7f6ff8c3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T01:59:33.764 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:33.763+0000 7f85a6012040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T01:59:33.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f257400ac30 con 0x7f257c0b3330 2024-03-25T01:59:33.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f2574025b80 con 0x7f257c0b3330 2024-03-25T01:59:33.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 10 client.4156.objecter ms_dispatch 0x7f257c000b90 osd_map(9..9 src has 1..9) 2024-03-25T01:59:33.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T01:59:33.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T01:59:33.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T01:59:33.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2574012420 con 0x7f257c0b3330 2024-03-25T01:59:33.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:33.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.857+0000 7f256affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:33.881 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:33.880+0000 7f6ff8c3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T01:59:33.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.928+0000 7f2580d04640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "23e89d51-cb05-4ca8-a5f0-1bf573aad7be", "id": 7}] 2024-03-25T01:59:33.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.928+0000 7f2580d04640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "23e89d51-cb05-4ca8-a5f0-1bf573aad7be", "id": 7}] 2024-03-25T01:59:33.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.928+0000 7f2580d04640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:33.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:33.928+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd new", "uuid": "23e89d51-cb05-4ca8-a5f0-1bf573aad7be", "id": 7} v 0) -- 0x7f257c13d210 con 0x7f257c0b3330 2024-03-25T01:59:34.041 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:34.039+0000 7f85a6012040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T01:59:34.096 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:34.095+0000 7f85a6012040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T01:59:34.218 INFO:tasks.ceph.mgr.y.smithi083.stderr:2024-03-25T01:59:34.217+0000 7f85a6012040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T01:59:34.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.283+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 2) ==== 45281+0+0 (secure 0 0 0) 0x7f2574025520 con 0x7f257c0b3330 2024-03-25T01:59:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.530+0000 7f256affd640 1 -- 172.21.15.153:0/3461709967 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23e89d51-cb05-4ca8-a5f0-1bf573aad7be", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f25740055f0 con 0x7f257c0b3330 2024-03-25T01:59:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.530+0000 7f256affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "23e89d51-cb05-4ca8-a5f0-1bf573aad7be", "id": 7}] 2024-03-25T01:59:34.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.530+0000 7f256affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:34.532 INFO:teuthology.orchestra.run.smithi153.stdout:7 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.530+0000 7f2580d04640 10 librados: watch_flush enter 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.530+0000 7f2580d04640 10 librados: watch_flush exit 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.530+0000 7f2580d04640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.530+0000 7f2580d04640 10 client.4156.objecter successfully canceled tick 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 10 monclient: shutdown 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 msgr2=0x7f257c1ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:34.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1ceab0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f257400c7a0 tx=0x7f257400cc60 comp rx=0 tx=0).stop 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 shutdown_connections 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f257c0b3330 0x7f257c1ceab0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f257c0a9f00 0x7f257c1cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 --2- 172.21.15.153:0/3461709967 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f257c0a9b20 0x7f257c1cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 >> 172.21.15.153:0/3461709967 conn(0x7f257c0b89b0 msgr2=0x7f257c0b8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 shutdown_connections 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 -- 172.21.15.153:0/3461709967 wait complete. 2024-03-25T01:59:34.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-25T01:59:34.531+0000 7f2580d04640 1 librados: shutdown 2024-03-25T01:59:34.544 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.544 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T01:59:34.544 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T01:59:34.546 INFO:tasks.ceph.osd.0:Started 2024-03-25T01:59:34.546 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.546 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T01:59:34.547 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T01:59:34.548 INFO:tasks.ceph.osd.1:Started 2024-03-25T01:59:34.549 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.549 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T01:59:34.549 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T01:59:34.551 INFO:tasks.ceph.osd.2:Started 2024-03-25T01:59:34.551 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.551 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T01:59:34.551 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T01:59:34.553 INFO:tasks.ceph.osd.3:Started 2024-03-25T01:59:34.554 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.554 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T01:59:34.554 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T01:59:34.556 INFO:tasks.ceph.osd.4:Started 2024-03-25T01:59:34.556 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.556 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T01:59:34.557 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T01:59:34.558 INFO:tasks.ceph.osd.5:Started 2024-03-25T01:59:34.558 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.559 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T01:59:34.559 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T01:59:34.561 INFO:tasks.ceph.osd.6:Started 2024-03-25T01:59:34.561 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T01:59:34.561 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T01:59:34.561 DEBUG:teuthology.orchestra.run.smithi153:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T01:59:34.563 INFO:tasks.ceph.osd.7:Started 2024-03-25T01:59:34.563 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T01:59:34.701 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.699+0000 7f6fbeb82640 10 monclient: get_monmap_and_config 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.699+0000 7f6fbeb82640 10 monclient: build_initial_monmap 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.699+0000 7f6fbeb82640 10 monclient: monmap: 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:59:34.701236+0000 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:59:34.701236+0000 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:34.702 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:34.703 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:34.703 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:34.703 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.700+0000 7f6fbeb82640 1 Processor -- start 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.700+0000 7f6fbeb82640 1 -- start start 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.700+0000 7f6fbeb82640 10 monclient: init 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fb80b04a0 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6fb80b2df0 0x7f6fb8140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb80b2df0 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fb8147350 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fb80b2410 con 0x7f6fb80b04a0 2024-03-25T01:59:34.705 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.701+0000 7f6fbeb82640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fb80b26f0 con 0x7f6fb8147350 2024-03-25T01:59:34.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.702+0000 7f6fbeb82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:34.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.702+0000 7f6fbeb82640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6fb80b29d0 con 0x7f6fb80b2df0 2024-03-25T01:59:34.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.702+0000 7f6fbeb82640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:34.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.702+0000 7f6fbeb82640 10 monclient(hunting): authenticate will time out at 1293.891979s 2024-03-25T01:59:34.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.702+0000 7f6fbd11f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6fb8147350 0x7f6fb814daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:52970/0 (socket says 172.21.15.83:52970) 2024-03-25T01:59:34.706 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.702+0000 7f6fbd11f640 1 -- 172.21.15.83:0/2531195451 learned_addr learned my addr 172.21.15.83:0/2531195451 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T01:59:34.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.732+0000 7f6fb7fff640 1 --2- 172.21.15.83:0/2531195451 >> v2:172.21.15.83:3300/0 conn(0x7f6fb80b2df0 0x7f6fb8140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:34.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.753+0000 7f6fb7fff640 10 monclient(hunting): get_auth_request con 0x7f6fb80b2df0 auth_method 0 2024-03-25T01:59:34.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.753+0000 7f6fb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:34.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.753+0000 7f6fb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:34.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.753+0000 7f6fb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:34.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.788+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:34.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.788+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:34.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.788+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 10 monclient(hunting): handle_auth_done global_id 4164 payload 274 2024-03-25T01:59:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 1 -- 172.21.15.83:0/2531195451 >> v1:172.21.15.153:6789/0 conn(0x7f6fb8147350 legacy=0x7f6fb814daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 1 -- 172.21.15.83:0/2531195451 >> v1:172.21.15.83:6789/0 conn(0x7f6fb80b04a0 legacy=0x7f6fb8146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 10 monclient: _finish_hunting 0 2024-03-25T01:59:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 1 monclient: found mon.noname-a 2024-03-25T01:59:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:34.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:34.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.795+0000 7f6fb7fff640 1 -- 172.21.15.83:0/2531195451 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb80b3380 con 0x7f6fb80b2df0 2024-03-25T01:59:34.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.805+0000 7f6fb7fff640 1 --2- 172.21.15.83:0/2531195451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 0x7f6fb8140400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40097d0 tx=0x7f6fa402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48772123e8665156 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:34.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.951+0000 7f6fb77fe640 1 -- 172.21.15.83:0/2531195451 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4034070 con 0x7f6fb80b2df0 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.951+0000 7f6fb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.951+0000 7f6fb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.951+0000 7f6fb77fe640 10 monclient: dump: 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:34.954 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.951+0000 7f6fb77fe640 10 monclient: _finish_auth 0 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.951+0000 7f6fb77fe640 10 monclient: _check_auth_tickets 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.951+0000 7f6fb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fb77fe640 1 -- 172.21.15.83:0/2531195451 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa4030610 con 0x7f6fb80b2df0 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fb77fe640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fb77fe640 1 -- 172.21.15.83:0/2531195451 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa40309e0 con 0x7f6fb80b2df0 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fb77fe640 10 monclient: dump: 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:34.955 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fbeb82640 5 monclient: authenticate success, global_id 4164 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fbeb82640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:34.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fbeb82640 4 set_mon_vals no callback set 2024-03-25T01:59:34.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fbeb82640 10 monclient: shutdown 2024-03-25T01:59:34.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fbeb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:34.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fbeb82640 1 -- 172.21.15.83:0/2531195451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 msgr2=0x7f6fb8140400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:34.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.952+0000 7f6fbeb82640 1 --2- 172.21.15.83:0/2531195451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 0x7f6fb8140400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40097d0 tx=0x7f6fa402b960 comp rx=0 tx=0).stop 2024-03-25T01:59:34.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.953+0000 7f6fbeb82640 1 -- 172.21.15.83:0/2531195451 shutdown_connections 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.953+0000 7f6fbeb82640 1 --2- 172.21.15.83:0/2531195451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 0x7f6fb8140400 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.953+0000 7f6fbeb82640 1 -- 172.21.15.83:0/2531195451 >> 172.21.15.83:0/2531195451 conn(0x7f6fb80b8a00 msgr2=0x7f6fb80b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.953+0000 7f6fbeb82640 1 -- 172.21.15.83:0/2531195451 shutdown_connections 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.953+0000 7f6fbeb82640 1 -- 172.21.15.83:0/2531195451 wait complete. 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.954+0000 7f6fbeb82640 10 monclient: build_initial_monmap 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.954+0000 7f6fbeb82640 10 monclient: monmap: 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:34.962 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.954+0000 7f6fbeb82640 1 librados: starting msgr at 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.954+0000 7f6fbeb82640 1 librados: starting objecter 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.954+0000 7f6fbeb82640 1 Processor -- start 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.954+0000 7f6fbeb82640 1 -- start start 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.955+0000 7f6fbeb82640 1 librados: setting wanted keys 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.955+0000 7f6fbeb82640 1 librados: calling monclient init 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.955+0000 7f6fbeb82640 10 monclient: init 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T01:59:34.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fb80b04a0 0x7f6fb81dbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 10 monclient(hunting): picked mon.noname-b con 0x7f6fb80b04a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 0x7f6fb80a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 10 monclient(hunting): picked mon.noname-a con 0x7f6fb80b2df0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6fb8147350 0x7f6fb80a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 10 monclient(hunting): picked mon.noname-c con 0x7f6fb8147350 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.956+0000 7f6fbeb82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbeb82640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb8140ac0 con 0x7f6fb80b2df0 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbeb82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbeb82640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6fb8140940 con 0x7f6fb80b04a0 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbeb82640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbeb82640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f6fb8140c40 con 0x7f6fb8147350 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbeb82640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbeb82640 10 monclient(hunting): authenticate will time out at 1294.147285s 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbc91e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fb80b04a0 0x7f6fb81dbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:34.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.957+0000 7f6fbd11f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6fb8147350 0x7f6fb80a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6fb8147350 0x7f6fb80a9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50560/0 (socket says 172.21.15.83:50560) 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 1 -- 172.21.15.83:0/970334847 learned_addr learned my addr 172.21.15.83:0/970334847 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 10 monclient(hunting): get_auth_request con 0x7f6fb8147350 auth_method 0 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbc91e640 10 monclient(hunting): get_auth_request con 0x7f6fb80b04a0 auth_method 0 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbc91e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbc91e640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbc91e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:34.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.958+0000 7f6fbd11f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:34.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.959+0000 7f6fbc91e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:34.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.959+0000 7f6fbc91e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:34.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.959+0000 7f6fbc91e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:34.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 10 monclient(hunting): handle_auth_done global_id 4145 payload 824 2024-03-25T01:59:34.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 1 -- 172.21.15.83:0/970334847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fb80b04a0 msgr2=0x7f6fb81dbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:34.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 1 --2- 172.21.15.83:0/970334847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fb80b04a0 0x7f6fb81dbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:34.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 1 -- 172.21.15.83:0/970334847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 msgr2=0x7f6fb80a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 1 --2- 172.21.15.83:0/970334847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 0x7f6fb80a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 10 monclient: _finish_hunting 0 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 1 monclient: found mon.noname-c 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.964+0000 7f6fbd11f640 1 -- 172.21.15.83:0/970334847 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fb81dc840 con 0x7f6fb8147350 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.967+0000 7f6fbd11f640 1 --2- 172.21.15.83:0/970334847 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6fb8147350 0x7f6fb80a9a20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000c9f0 tx=0x7f6fb000ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.967+0000 7f6fb57fa640 10 client.?.objecter ms_handle_connect 0x7f6fb8147350 2024-03-25T01:59:34.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:34.967+0000 7f6fb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:35.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 1 -- 172.21.15.83:0/970334847 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0017070 con 0x7f6fb8147350 2024-03-25T01:59:35.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:35.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:35.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: dump: 2024-03-25T01:59:35.040 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:35.040 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:35.040 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: _finish_auth 0 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: _check_auth_tickets 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 1 -- 172.21.15.83:0/970334847 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb000ab70 con 0x7f6fb8147350 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 1 -- 172.21.15.83:0/970334847 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0012400 con 0x7f6fb8147350 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:35.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb57fa640 10 monclient: dump: 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 5 monclient: authenticate success, global_id 4145 2024-03-25T01:59:35.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 10 monclient: _renew_subs 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fb81dc4d0 con 0x7f6fb8147350 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fb6ffd640 4 set_mon_vals no callback set 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 10 monclient: _renew_subs 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fb80a9f60 con 0x7f6fb8147350 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.036+0000 7f6fbeb82640 1 librados: init done 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.038+0000 7f6fbeb82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.038+0000 7f6fbeb82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.038+0000 7f6fbeb82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:35.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.038+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fb8075810 con 0x7f6fb8147350 2024-03-25T01:59:35.061 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.059+0000 7f6fb57fa640 1 -- 172.21.15.83:0/970334847 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 2) ==== 45281+0+0 (secure 0 0 0) 0x7f6fb001a070 con 0x7f6fb8147350 2024-03-25T01:59:35.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.063+0000 7f6fb57fa640 1 -- 172.21.15.83:0/970334847 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f6fb0011c10 con 0x7f6fb8147350 2024-03-25T01:59:35.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.063+0000 7f6fb57fa640 10 client.4145.objecter ms_dispatch 0x7f6fb8000b90 osd_map(10..10 src has 1..10) 2024-03-25T01:59:35.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.063+0000 7f6fb57fa640 3 client.4145.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T01:59:35.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.063+0000 7f6fb57fa640 3 client.4145.objecter handle_osd_map decoding full epoch 10 2024-03-25T01:59:35.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.063+0000 7f6fb57fa640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-25T01:59:35.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.234+0000 7f6fb57fa640 1 -- 172.21.15.83:0/970334847 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6fb001a360 con 0x7f6fb8147350 2024-03-25T01:59:35.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.234+0000 7f6fb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:35.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.234+0000 7f6fb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.355+0000 7f6fbeb82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.355+0000 7f6fbeb82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.355+0000 7f6fbeb82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:35.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.355+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6fb81dca10 con 0x7f6fb8147350 2024-03-25T01:59:35.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.578+0000 7f6fb57fa640 1 -- 172.21.15.83:0/970334847 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f6fb001bda0 con 0x7f6fb8147350 2024-03-25T01:59:35.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.578+0000 7f6fb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:35.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.578+0000 7f6fb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:35.581 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T01:59:35.581 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":10,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T01:59:34.053585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T01:59:35.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 10 librados: watch_flush enter 2024-03-25T01:59:35.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 10 librados: watch_flush exit 2024-03-25T01:59:35.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-25T01:59:35.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 10 client.4145.objecter successfully canceled tick 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 10 monclient: shutdown 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6fb8147350 msgr2=0x7f6fb80a9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.579+0000 7f6fbeb82640 1 --2- 172.21.15.83:0/970334847 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6fb8147350 0x7f6fb80a9a20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6fb000c9f0 tx=0x7f6fb000ceb0 comp rx=0 tx=0).stop 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 shutdown_connections 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 --2- 172.21.15.83:0/970334847 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6fb8147350 0x7f6fb80a9a20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 --2- 172.21.15.83:0/970334847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6fb80b2df0 0x7f6fb80a94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 --2- 172.21.15.83:0/970334847 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6fb80b04a0 0x7f6fb81dbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 >> 172.21.15.83:0/970334847 conn(0x7f6fb80b8a00 msgr2=0x7f6fb8146b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 shutdown_connections 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 -- 172.21.15.83:0/970334847 wait complete. 2024-03-25T01:59:35.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T01:59:35.580+0000 7f6fbeb82640 1 librados: shutdown 2024-03-25T01:59:35.597 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T01:59:35.598 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T01:59:36.143 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:36.142+0000 7f6f997d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T01:59:36.156 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T01:59:36.155+0000 7f6f997d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T01:59:36.404 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.543+0000 7f05233b2640 10 monclient: get_monmap_and_config 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.543+0000 7f05233b2640 10 monclient: build_initial_monmap 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.543+0000 7f05233b2640 10 monclient: monmap: 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:59:36.544662+0000 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:59:36.544662+0000 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:36.545 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:36.546 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:36.546 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:36.546 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:36.546 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:36.546 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:36.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.544+0000 7f05233b2640 1 Processor -- start 2024-03-25T01:59:36.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.544+0000 7f05233b2640 1 -- start start 2024-03-25T01:59:36.549 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.544+0000 7f05233b2640 10 monclient: init 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c0b5a10 0x7f051c143750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient(hunting): picked mon.noname-b con 0x7f051c0b5a10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f051c143c90 0x7f051c146c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient(hunting): picked mon.noname-a con 0x7f051c143c90 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f051c147160 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f051c1440d0 con 0x7f051c147160 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f051c0b5df0 con 0x7f051c143c90 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f051c0ab9c0 con 0x7f051c0b5a10 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:36.550 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.545+0000 7f05233b2640 10 monclient(hunting): authenticate will time out at 1295.735463s 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052094d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f051c143c90 0x7f051c146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052094d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f051c143c90 0x7f051c146c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42498/0 (socket says 172.21.15.83:42498) 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052094d640 1 -- 172.21.15.83:0/1115111360 learned_addr learned my addr 172.21.15.83:0/1115111360 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052094d640 10 monclient(hunting): get_auth_request con 0x7f051c143c90 auth_method 0 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052094d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052094d640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052094d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052114e640 1 --2- 172.21.15.83:0/1115111360 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c0b5a10 0x7f051c143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052114e640 10 monclient(hunting): get_auth_request con 0x7f051c0b5a10 auth_method 0 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052114e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052114e640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.546+0000 7f052114e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.547+0000 7f052094d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.547+0000 7f052094d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:36.551 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.547+0000 7f052094d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:36.552 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.547+0000 7f052114e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:36.552 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.547+0000 7f052114e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:36.552 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.547+0000 7f052114e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 10 monclient(hunting): handle_auth_done global_id 4148 payload 274 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 1 -- 172.21.15.83:0/1115111360 >> v2:172.21.15.83:3300/0 conn(0x7f051c143c90 msgr2=0x7f051c146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 1 --2- 172.21.15.83:0/1115111360 >> v2:172.21.15.83:3300/0 conn(0x7f051c143c90 0x7f051c146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 1 -- 172.21.15.83:0/1115111360 >> v1:172.21.15.83:6789/0 conn(0x7f051c147160 legacy=0x7f051c0b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 10 monclient: _finish_hunting 0 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 1 monclient: found mon.noname-b 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T01:59:36.553 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.549+0000 7f052114e640 1 -- 172.21.15.83:0/1115111360 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051c0abbf0 con 0x7f051c0b5a10 2024-03-25T01:59:36.554 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.552+0000 7f052114e640 1 --2- 172.21.15.83:0/1115111360 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c0b5a10 0x7f051c143750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f05180097a0 tx=0x7f051802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3f201031a7c3dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.557+0000 7f0513fff640 1 -- 172.21.15.83:0/1115111360 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0518035020 con 0x7f051c0b5a10 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.557+0000 7f0513fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: dump: 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:36.562 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: _finish_auth 0 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: _check_auth_tickets 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f05233b2640 5 monclient: authenticate success, global_id 4148 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f05233b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 1 -- 172.21.15.83:0/1115111360 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0518031830 con 0x7f051c0b5a10 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f05233b2640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f05233b2640 4 set_mon_vals no callback set 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 1 -- 172.21.15.83:0/1115111360 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0518035020 con 0x7f051c0b5a10 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:36.563 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.558+0000 7f0513fff640 10 monclient: dump: 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.559+0000 7f05233b2640 10 monclient: shutdown 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.559+0000 7f05233b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:36.564 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.559+0000 7f05233b2640 1 -- 172.21.15.83:0/1115111360 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c0b5a10 msgr2=0x7f051c143750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:36.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.559+0000 7f05233b2640 1 --2- 172.21.15.83:0/1115111360 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c0b5a10 0x7f051c143750 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f05180097a0 tx=0x7f051802b8d0 comp rx=0 tx=0).stop 2024-03-25T01:59:36.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.560+0000 7f05233b2640 1 -- 172.21.15.83:0/1115111360 shutdown_connections 2024-03-25T01:59:36.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.560+0000 7f05233b2640 1 --2- 172.21.15.83:0/1115111360 >> v2:172.21.15.83:3300/0 conn(0x7f051c143c90 0x7f051c146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:36.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.560+0000 7f05233b2640 1 --2- 172.21.15.83:0/1115111360 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c0b5a10 0x7f051c143750 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:36.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.560+0000 7f05233b2640 1 -- 172.21.15.83:0/1115111360 >> 172.21.15.83:0/1115111360 conn(0x7f051c067460 msgr2=0x7f051c067870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:36.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.560+0000 7f05233b2640 1 -- 172.21.15.83:0/1115111360 shutdown_connections 2024-03-25T01:59:36.565 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.560+0000 7f05233b2640 1 -- 172.21.15.83:0/1115111360 wait complete. 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 10 monclient: build_initial_monmap 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 10 monclient: monmap: 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:36.571 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 1 librados: starting msgr at 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 1 librados: starting objecter 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 1 Processor -- start 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 1 -- start start 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 1 librados: setting wanted keys 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 1 librados: calling monclient init 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.561+0000 7f05233b2640 10 monclient: init 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0b5a10 0x7f051c0b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): picked mon.noname-b con 0x7f051c0b5a10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f051c143c90 0x7f051c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): picked mon.noname-a con 0x7f051c143c90 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:36.572 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c147160 0x7f051c13dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): picked mon.noname-c con 0x7f051c147160 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f051c1475d0 con 0x7f051c143c90 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f051c146790 con 0x7f051c0b5a10 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f051c143180 con 0x7f051c147160 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.562+0000 7f05233b2640 10 monclient(hunting): authenticate will time out at 1295.752921s 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052094d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f051c143c90 0x7f051c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052194f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c147160 0x7f051c13dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052094d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f051c143c90 0x7f051c0b1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42506/0 (socket says 172.21.15.83:42506) 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052094d640 1 -- 172.21.15.83:0/1504390399 learned_addr learned my addr 172.21.15.83:0/1504390399 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:36.573 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052094d640 10 monclient(hunting): get_auth_request con 0x7f051c143c90 auth_method 0 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052094d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052094d640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052094d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.563+0000 7f052114e640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0b5a10 0x7f051c0b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052194f640 10 monclient(hunting): get_auth_request con 0x7f051c147160 auth_method 0 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052194f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052194f640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052194f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052114e640 10 monclient(hunting): get_auth_request con 0x7f051c0b5a10 auth_method 0 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052114e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052114e640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052114e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052094d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:36.574 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052094d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052094d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052194f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052194f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.564+0000 7f052194f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.565+0000 7f052114e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.565+0000 7f052114e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.565+0000 7f052114e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 10 monclient(hunting): handle_auth_done global_id 4173 payload 824 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 1 -- 172.21.15.83:0/1504390399 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c147160 msgr2=0x7f051c13dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c147160 0x7f051c13dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 1 -- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0b5a10 msgr2=0x7f051c0b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0b5a10 0x7f051c0b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 10 monclient: _finish_hunting 0 2024-03-25T01:59:36.575 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 1 monclient: found mon.noname-a 2024-03-25T01:59:36.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:36.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:36.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.568+0000 7f052094d640 1 -- 172.21.15.83:0/1504390399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051c0b2320 con 0x7f051c143c90 2024-03-25T01:59:36.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.572+0000 7f052094d640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f051c143c90 0x7f051c0b1bb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f050c009fd0 tx=0x7f050c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:36.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.572+0000 7f0511ffb640 10 client.?.objecter ms_handle_connect 0x7f051c143c90 2024-03-25T01:59:36.576 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.572+0000 7f0511ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:36.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c017070 con 0x7f051c143c90 2024-03-25T01:59:36.579 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: dump: 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: _finish_auth 0 2024-03-25T01:59:36.580 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: _check_auth_tickets 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f050c00c940 con 0x7f051c143c90 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c00cd10 con 0x7f051c143c90 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f0511ffb640 10 monclient: dump: 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.583 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f05233b2640 5 monclient: authenticate success, global_id 4173 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f05233b2640 10 monclient: _renew_subs 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f05137fe640 4 set_mon_vals no callback set 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f05233b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.577+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051c121230 con 0x7f051c143c90 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.578+0000 7f05233b2640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.578+0000 7f05233b2640 10 monclient: _renew_subs 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.578+0000 7f05233b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:36.584 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.578+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f051c13e420 con 0x7f051c143c90 2024-03-25T01:59:36.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.578+0000 7f05233b2640 1 librados: init done 2024-03-25T01:59:36.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.579+0000 7f05233b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:36.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.579+0000 7f05233b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:36.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.579+0000 7f05233b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:36.585 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.579+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f051c121050 con 0x7f051c143c90 2024-03-25T01:59:36.998 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.996+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 3) ==== 90350+0+0 (secure 0 0 0) 0x7f050c01a070 con 0x7f051c143c90 2024-03-25T01:59:36.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.997+0000 7f0511ffb640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f050006d870 0x7f050006fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:36.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.997+0000 7f052114e640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f050006d870 0x7f050006fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:36.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.997+0000 7f052114e640 10 monclient: get_auth_request con 0x7f050006d870 auth_method 0 2024-03-25T01:59:36.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.998+0000 7f0511ffb640 10 client.4173.objecter ms_handle_connect 0x7f050006d870 2024-03-25T01:59:36.999 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:36.998+0000 7f052114e640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f050006d870 0x7f050006fd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0518036000 tx=0x7f0518008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:37.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f050c058210 con 0x7f051c143c90 2024-03-25T01:59:37.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 10 client.4173.objecter ms_dispatch 0x7f051c000b90 osd_map(10..10 src has 1..10) 2024-03-25T01:59:37.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T01:59:37.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T01:59:37.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T01:59:37.021 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f051c121050 con 0x7f051c143c90 2024-03-25T01:59:37.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:37.022 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.019+0000 7f0511ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:37.313 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.311+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7f050c055bf0 con 0x7f051c143c90 2024-03-25T01:59:37.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.383+0000 7f05233b2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:37.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.383+0000 7f05233b2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:37.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.383+0000 7f05233b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:37.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.383+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f051c0692f0 con 0x7f051c143c90 2024-03-25T01:59:37.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.454+0000 7f0511ffb640 1 -- 172.21.15.83:0/1504390399 <== mon.0 v2:172.21.15.83:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f050c07a380 con 0x7f051c143c90 2024-03-25T01:59:37.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.454+0000 7f0511ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:37.456 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.454+0000 7f0511ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:37.456 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T01:59:37.456 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T01:59:34.053585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T01:59:37.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 10 librados: watch_flush enter 2024-03-25T01:59:37.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 10 librados: watch_flush exit 2024-03-25T01:59:37.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T01:59:37.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 10 client.4173.objecter successfully canceled tick 2024-03-25T01:59:37.460 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f050006d870 msgr2=0x7f050006fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f050006d870 0x7f050006fd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0518036000 tx=0x7f0518008030 comp rx=0 tx=0).stop 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 10 monclient: shutdown 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f051c143c90 msgr2=0x7f051c0b1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f051c143c90 0x7f051c0b1bb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f050c009fd0 tx=0x7f050c00ed40 comp rx=0 tx=0).stop 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 shutdown_connections 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f050006d870 0x7f050006fd30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f051c147160 0x7f051c13dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f051c143c90 0x7f051c0b1bb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 --2- 172.21.15.83:0/1504390399 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f051c0b5a10 0x7f051c0b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 >> 172.21.15.83:0/1504390399 conn(0x7f051c067460 msgr2=0x7f051c067840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 shutdown_connections 2024-03-25T01:59:37.461 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.457+0000 7f05233b2640 1 -- 172.21.15.83:0/1504390399 wait complete. 2024-03-25T01:59:37.462 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:37.458+0000 7f05233b2640 1 librados: shutdown 2024-03-25T01:59:37.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T01:59:41.765 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:07.139 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:41.839 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:07.214 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:41.915 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:07.290 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:41.975 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:07.351 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:41.989 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:07.365 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.047 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:07.423 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.077 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:07.452 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.134 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:07.509 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.212 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:07.577 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.264 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:07.623 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.277 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:07.642 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.332 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:07.692 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.356 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:07.719 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.391 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T01:59:42.384+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:42.414 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:07.778 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.419 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:07.784 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.478 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:07.843 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.522 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T01:59:42.514+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:42.602 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T01:59:42.595+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:42.708 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T01:59:42.701+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:42.827 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T01:59:42.819+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:42.850 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:08.225 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:42.867 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T01:59:42.859+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:42.965 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T01:59:42.957+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:43.012 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:08.387 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.057 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T01:59:43.049+0000 5dc3480 -1 Falling back to public interface 2024-03-25T01:59:43.082 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:08.458 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.194 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:08.568 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.250 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:08.624 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.273 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:08.638 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.327 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:08.686 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.336 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:08.711 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.403 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:08.779 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.451 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:08.814 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.475 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:08.850 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.554 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:08.919 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.579 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:08.954 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.666 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:09.031 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.670 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.045 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.718 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:09.077 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.734 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:09.110 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.770 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.146 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.839 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:09.203 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.866 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:09.241 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.942 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:09.307 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.958 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:09.323 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:43.965 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.340 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.006 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:09.365 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.022 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:09.398 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.037 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.413 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.153 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:09.528 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.171 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:09.534 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.233 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:09.598 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.258 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.633 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.269 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:09.628 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.279 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T01:59:44.285 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:09.647 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.308 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.684 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.317 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:09.693 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.364+0000 7f35a137a640 10 monclient: get_monmap_and_config 2024-03-25T01:59:44.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.364+0000 7f35a137a640 10 monclient: build_initial_monmap 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.364+0000 7f35a137a640 10 monclient: monmap: 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:59:44.366316+0000 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:59:44.366316+0000 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:44.367 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:44.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.365+0000 7f35a137a640 1 Processor -- start 2024-03-25T01:59:44.369 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.365+0000 7f35a137a640 1 -- start start 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.365+0000 7f35a137a640 10 monclient: init 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f359c06e860 addr v1:172.21.15.83:6789/0 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f359c072fe0 0x7f359c071e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient(hunting): picked mon.noname-a con 0x7f359c072fe0 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c072360 0x7f359c14db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient(hunting): picked mon.noname-b con 0x7f359c072360 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359c0b14d0 con 0x7f359c06e860 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f359c06ec40 con 0x7f359c072fe0 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f359c140f40 con 0x7f359c072360 2024-03-25T01:59:44.370 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f35a137a640 10 monclient(hunting): authenticate will time out at 1303.556100s 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f359a7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f359c072fe0 0x7f359c071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.366+0000 7f359b7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c072360 0x7f359c14db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f359c072fe0 0x7f359c071e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41924/0 (socket says 172.21.15.83:41924) 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 1 -- 172.21.15.83:0/1982193232 learned_addr learned my addr 172.21.15.83:0/1982193232 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 10 monclient(hunting): get_auth_request con 0x7f359c072fe0 auth_method 0 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359b7fe640 10 monclient(hunting): get_auth_request con 0x7f359c072360 auth_method 0 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:44.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:44.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:44.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:44.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:44.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:44.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.367+0000 7f359b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:44.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 10 monclient(hunting): handle_auth_done global_id 4157 payload 274 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 1 -- 172.21.15.83:0/1982193232 >> v2:172.21.15.83:3300/0 conn(0x7f359c072fe0 msgr2=0x7f359c071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 1 --2- 172.21.15.83:0/1982193232 >> v2:172.21.15.83:3300/0 conn(0x7f359c072fe0 0x7f359c071e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 1 -- 172.21.15.83:0/1982193232 >> v1:172.21.15.83:6789/0 conn(0x7f359c06e860 legacy=0x7f359c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 10 monclient: _finish_hunting 0 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 1 monclient: found mon.noname-b 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.369+0000 7f359b7fe640 1 -- 172.21.15.83:0/1982193232 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c071720 con 0x7f359c072360 2024-03-25T01:59:44.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.371+0000 7f359b7fe640 1 --2- 172.21.15.83:0/1982193232 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c072360 0x7f359c14db00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f358c009d90 tx=0x7f358c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221410d61e6b061d server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:44.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 1 -- 172.21.15.83:0/1982193232 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c039070 con 0x7f359c072360 2024-03-25T01:59:44.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:44.376 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: dump: 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: _finish_auth 0 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: _check_auth_tickets 2024-03-25T01:59:44.377 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 1 -- 172.21.15.83:0/1982193232 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f358c02cbc0 con 0x7f359c072360 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f35a137a640 5 monclient: authenticate success, global_id 4157 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f35a137a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f35a137a640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f35a137a640 4 set_mon_vals no callback set 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 1 -- 172.21.15.83:0/1982193232 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c034440 con 0x7f359c072360 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.374+0000 7f3599ffb640 10 monclient: dump: 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.378 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.379 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:44.379 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:44.379 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:44.379 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:44.379 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:44.379 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 10 monclient: shutdown 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 -- 172.21.15.83:0/1982193232 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c072360 msgr2=0x7f359c14db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 --2- 172.21.15.83:0/1982193232 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c072360 0x7f359c14db00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f358c009d90 tx=0x7f358c02be10 comp rx=0 tx=0).stop 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 -- 172.21.15.83:0/1982193232 shutdown_connections 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 --2- 172.21.15.83:0/1982193232 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c072360 0x7f359c14db00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 --2- 172.21.15.83:0/1982193232 >> v2:172.21.15.83:3300/0 conn(0x7f359c072fe0 0x7f359c071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 -- 172.21.15.83:0/1982193232 >> 172.21.15.83:0/1982193232 conn(0x7f359c09ba50 msgr2=0x7f359c06dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:44.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 -- 172.21.15.83:0/1982193232 shutdown_connections 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.375+0000 7f35a137a640 1 -- 172.21.15.83:0/1982193232 wait complete. 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 10 monclient: build_initial_monmap 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 10 monclient: monmap: 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:44.384 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 1 librados: starting msgr at 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 1 librados: starting objecter 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 1 Processor -- start 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 1 -- start start 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 1 librados: setting wanted keys 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 1 librados: calling monclient init 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.376+0000 7f35a137a640 10 monclient: init 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c06e860 0x7f359c0b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): picked mon.noname-c con 0x7f359c06e860 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f359c072360 0x7f359c1dbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): picked mon.noname-a con 0x7f359c072360 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:44.385 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f359c072fe0 0x7f359c1445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): picked mon.noname-b con 0x7f359c072fe0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f359c0b1ec0 con 0x7f359c072360 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f359c0b3010 con 0x7f359c072fe0 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f359c0b2040 con 0x7f359c06e860 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f35a137a640 10 monclient(hunting): authenticate will time out at 1303.567364s 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f359affd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c06e860 0x7f359c0b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.377+0000 7f359a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f359c072360 0x7f359c1dbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f359c072fe0 0x7f359c1445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c06e860 0x7f359c0b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54350/0 (socket says 172.21.15.83:54350) 2024-03-25T01:59:44.386 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 1 -- 172.21.15.83:0/2018951680 learned_addr learned my addr 172.21.15.83:0/2018951680 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 10 monclient(hunting): get_auth_request con 0x7f359c06e860 auth_method 0 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359a7fc640 10 monclient(hunting): get_auth_request con 0x7f359c072360 auth_method 0 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359b7fe640 10 monclient(hunting): get_auth_request con 0x7f359c072fe0 auth_method 0 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:44.387 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.378+0000 7f359a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.379+0000 7f359b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.379+0000 7f359b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.379+0000 7f359b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 1 -- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f359c072fe0 msgr2=0x7f359c1445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f359c072fe0 0x7f359c1445d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 1 -- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f359c072360 msgr2=0x7f359c1dbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f359c072360 0x7f359c1dbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 10 monclient: _finish_hunting 0 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 1 monclient: found mon.noname-c 2024-03-25T01:59:44.388 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:44.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T01:59:44.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359affd640 1 -- 172.21.15.83:0/2018951680 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c142250 con 0x7f359c06e860 2024-03-25T01:59:44.389 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.381+0000 7f359a7fc640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f359c072360 0x7f359c1dbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T01:59:44.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.383+0000 7f359affd640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c06e860 0x7f359c0b6c60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f359400c430 tx=0x7f359400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:44.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.383+0000 7f35837fe640 10 client.?.objecter ms_handle_connect 0x7f359c06e860 2024-03-25T01:59:44.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.383+0000 7f35837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 1 -- 172.21.15.83:0/2018951680 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35940052d0 con 0x7f359c06e860 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 10 monclient: dump: 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:44.391 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 10 monclient: _finish_auth 0 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 10 monclient: _check_auth_tickets 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35837fe640 1 -- 172.21.15.83:0/2018951680 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3594005490 con 0x7f359c06e860 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35a137a640 5 monclient: authenticate success, global_id 4160 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.386+0000 7f35a137a640 10 monclient: _renew_subs 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35a137a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359c142530 con 0x7f359c06e860 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35837fe640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35837fe640 1 -- 172.21.15.83:0/2018951680 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594007260 con 0x7f359c06e860 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35837fe640 10 monclient: dump: 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:44.392 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35997fa640 4 set_mon_vals no callback set 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35a137a640 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35a137a640 10 monclient: _renew_subs 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35a137a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f359c147500 con 0x7f359c06e860 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.387+0000 7f35a137a640 1 librados: init done 2024-03-25T01:59:44.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.388+0000 7f35a137a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:44.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.388+0000 7f35a137a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:44.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.388+0000 7f35a137a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:44.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.388+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f359c04f940 con 0x7f359c06e860 2024-03-25T01:59:44.444 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:09.819 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.457 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:09.820 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.498 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:09.863 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.538 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:09.897 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.545 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:09.920 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.570 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:09.936 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.573 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:09.949 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.608 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:09.984 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.640 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.638+0000 7f35837fe640 1 -- 172.21.15.83:0/2018951680 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7f3594007400 con 0x7f359c06e860 2024-03-25T01:59:44.640 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.638+0000 7f35837fe640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f358806d990 0x7f358806fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:44.641 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.639+0000 7f359a7fc640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f358806d990 0x7f358806fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:44.641 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.639+0000 7f359a7fc640 10 monclient: get_auth_request con 0x7f358806d990 auth_method 0 2024-03-25T01:59:44.641 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.639+0000 7f35837fe640 10 client.4160.objecter ms_handle_connect 0x7f358806d990 2024-03-25T01:59:44.641 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.639+0000 7f359a7fc640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f358806d990 0x7f358806fe50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f359c0b69c0 tx=0x7f3590009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:44.655 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 1 -- 172.21.15.83:0/2018951680 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f3594056d40 con 0x7f359c06e860 2024-03-25T01:59:44.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 10 client.4160.objecter ms_dispatch 0x7f359c000b90 osd_map(10..10 src has 1..10) 2024-03-25T01:59:44.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 3 client.4160.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T01:59:44.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 3 client.4160.objecter handle_osd_map decoding full epoch 10 2024-03-25T01:59:44.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 20 client.4160.objecter dump_active .. 0 homeless 2024-03-25T01:59:44.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 1 -- 172.21.15.83:0/2018951680 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3594058ca0 con 0x7f359c06e860 2024-03-25T01:59:44.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:44.656 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.654+0000 7f35837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:44.737 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:10.112 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.749 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:10.113 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.771 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:10.136 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.803 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.162 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.836 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.834+0000 7f35a137a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:44.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.834+0000 7f35a137a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:44.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.835+0000 7f35a137a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T01:59:44.837 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.835+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f359c0b1ca0 con 0x7f359c06e860 2024-03-25T01:59:44.839 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:10.214 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.843 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:10.219 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.849+0000 7f35837fe640 1 -- 172.21.15.83:0/2018951680 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f359405cd40 con 0x7f359c06e860 2024-03-25T01:59:44.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.849+0000 7f35837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:44.851 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.849+0000 7f35837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:44.852 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T01:59:44.852 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T01:59:34.053585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T01:59:44.854 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 10 librados: watch_flush enter 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 10 librados: watch_flush exit 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 20 client.4160.objecter shutdown clearing up homeless session... 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 10 client.4160.objecter successfully canceled tick 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f358806d990 msgr2=0x7f358806fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f358806d990 0x7f358806fe50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f359c0b69c0 tx=0x7f3590009240 comp rx=0 tx=0).stop 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 10 monclient: shutdown 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c06e860 msgr2=0x7f359c0b6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c06e860 0x7f359c0b6c60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f359400c430 tx=0x7f359400c8f0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 shutdown_connections 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f358806d990 0x7f358806fe50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f359c072fe0 0x7f359c1445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f359c072360 0x7f359c1dbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.855 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 --2- 172.21.15.83:0/2018951680 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f359c06e860 0x7f359c0b6c60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:44.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 >> 172.21.15.83:0/2018951680 conn(0x7f359c09ba50 msgr2=0x7f359c0ac1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:44.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 shutdown_connections 2024-03-25T01:59:44.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 -- 172.21.15.83:0/2018951680 wait complete. 2024-03-25T01:59:44.856 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:44.852+0000 7f35a137a640 1 librados: shutdown 2024-03-25T01:59:44.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T01:59:44.867 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:10.232 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:44.895 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:10.271 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.033 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:10.408 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.045 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:10.410 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.050 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:10.413 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.070 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.429 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.108 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:10.484 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.129 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:10.504 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.156 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:10.521 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.187 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:10.563 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.313 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:10.678 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.320 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:10.695 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.341 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.700 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.349 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:10.713 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.373 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:10.749 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.423 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:10.798 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.443 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:10.808 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.477 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:10.853 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.577 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:10.942 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.616 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:10.975 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.640 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:11.004 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.730 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:11.095 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.870 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:11.245 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.884 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:11.261 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.912 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:11.287 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.920 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:11.296 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.951 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:11.315 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:45.966 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:11.341 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.000 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:11.375 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.029 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:11.394 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.033 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:11.408 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.064 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:11.440 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.071 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:11.436 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.102 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:11.467 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.126 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:11.485 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.157 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:11.516 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.516 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:11.879 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.547 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:11.911 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.578 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:11.943 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:46.609 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:11.974 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:50.924 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:16.300 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:50.951 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:16.326 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:51.000 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:16.376 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:51.062 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:16.437 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:51.520 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:16.879 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:51.671 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T01:59:51.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.764+0000 7f3f90e84640 10 monclient: get_monmap_and_config 2024-03-25T01:59:51.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.764+0000 7f3f90e84640 10 monclient: build_initial_monmap 2024-03-25T01:59:51.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.764+0000 7f3f90e84640 10 monclient: monmap: 2024-03-25T01:59:51.766 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:59:51.765693+0000 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:59:51.765693+0000 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.764+0000 7f3f90e84640 1 Processor -- start 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.764+0000 7f3f90e84640 1 -- start start 2024-03-25T01:59:51.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.764+0000 7f3f90e84640 10 monclient: init 2024-03-25T01:59:51.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:51.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-03-25T01:59:51.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c0b38e0 0x7f3f8c140350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:51.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient(hunting): picked mon.noname-b con 0x7f3f8c0b38e0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3f8c140890 0x7f3f8c146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient(hunting): picked mon.noname-a con 0x7f3f8c140890 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f8c146bc0 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8c0b3300 con 0x7f3f8c146bc0 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3f8c0ab060 con 0x7f3f8c140890 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8c0abed0 con 0x7f3f8c0b38e0 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.765+0000 7f3f90e84640 10 monclient(hunting): authenticate will time out at 1310.955855s 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f89d9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3f8c140890 0x7f3f8c146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f8a59c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c0b38e0 0x7f3f8c140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:51.772 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f8ad9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3f8c146bc0 0x7f3f8c143120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:39906/0 (socket says 172.21.15.83:39906) 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f8ad9d640 1 -- 172.21.15.83:0/320695674 learned_addr learned my addr 172.21.15.83:0/320695674 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f89d9b640 10 monclient(hunting): get_auth_request con 0x7f3f8c140890 auth_method 0 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.766+0000 7f3f8a59c640 10 monclient(hunting): get_auth_request con 0x7f3f8c0b38e0 auth_method 0 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.767+0000 7f3f8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.767+0000 7f3f8a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.767+0000 7f3f8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.767+0000 7f3f89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.767+0000 7f3f89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.767+0000 7f3f89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.767+0000 7f3f8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:51.773 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.768+0000 7f3f8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.768+0000 7f3f8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f8959a640 1 -- 172.21.15.83:0/320695674 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 92569993 0 0) 0x7f3f8c0b3300 con 0x7f3f8c146bc0 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f8959a640 10 monclient(hunting): _negotiate 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f8959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f8959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f8959a640 10 monclient(hunting): my global_id is 4213 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f8959a640 1 -- 172.21.15.83:0/320695674 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f74003520 con 0x7f3f8c146bc0 2024-03-25T01:59:51.774 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 1 -- 172.21.15.83:0/320695674 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c0b38e0 msgr2=0x7f3f8c140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 1 --2- 172.21.15.83:0/320695674 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c0b38e0 0x7f3f8c140350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 1 -- 172.21.15.83:0/320695674 >> v1:172.21.15.153:6789/0 conn(0x7f3f8c146bc0 legacy=0x7f3f8c143120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 10 monclient: _finish_hunting 0 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 1 monclient: found mon.noname-a 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.769+0000 7f3f89d9b640 1 -- 172.21.15.83:0/320695674 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c0ac1b0 con 0x7f3f8c140890 2024-03-25T01:59:51.775 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.771+0000 7f3f89d9b640 1 --2- 172.21.15.83:0/320695674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c140890 0x7f3f8c146680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3f800097a0 tx=0x7f3f8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa61bf34651cf5a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:51.776 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 1 -- 172.21.15.83:0/320695674 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80035020 con 0x7f3f8c140890 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: dump: 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:51.777 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: _finish_auth 0 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: _check_auth_tickets 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 1 -- 172.21.15.83:0/320695674 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f80031830 con 0x7f3f8c140890 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 1 -- 172.21.15.83:0/320695674 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80031c00 con 0x7f3f8c140890 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f8959a640 10 monclient: dump: 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:51.778 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:51.779 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:51.779 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:51.779 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:51.779 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:51.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f90e84640 5 monclient: authenticate success, global_id 4215 2024-03-25T01:59:51.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f90e84640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:51.779 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.775+0000 7f3f90e84640 4 set_mon_vals no callback set 2024-03-25T01:59:51.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 10 monclient: shutdown 2024-03-25T01:59:51.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:51.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 -- 172.21.15.83:0/320695674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c140890 msgr2=0x7f3f8c146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:51.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 --2- 172.21.15.83:0/320695674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c140890 0x7f3f8c146680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3f800097a0 tx=0x7f3f8002b8d0 comp rx=0 tx=0).stop 2024-03-25T01:59:51.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 -- 172.21.15.83:0/320695674 shutdown_connections 2024-03-25T01:59:51.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 --2- 172.21.15.83:0/320695674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c140890 0x7f3f8c146680 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:51.783 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 --2- 172.21.15.83:0/320695674 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c0b38e0 0x7f3f8c140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 -- 172.21.15.83:0/320695674 >> 172.21.15.83:0/320695674 conn(0x7f3f8c067460 msgr2=0x7f3f8c067870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 -- 172.21.15.83:0/320695674 shutdown_connections 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.776+0000 7f3f90e84640 1 -- 172.21.15.83:0/320695674 wait complete. 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.777+0000 7f3f90e84640 10 monclient: build_initial_monmap 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.777+0000 7f3f90e84640 10 monclient: monmap: 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:51.784 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.777+0000 7f3f90e84640 1 librados: starting msgr at 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.777+0000 7f3f90e84640 1 librados: starting objecter 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 Processor -- start 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 -- start start 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 librados: setting wanted keys 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 librados: calling monclient init 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient: init 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): picked mon.noname-b con 0x7f3f8c0b38e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c140890 0x7f3f8c0b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): picked mon.noname-c con 0x7f3f8c140890 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c146bc0 0x7f3f8c1d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:51.785 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): picked mon.noname-a con 0x7f3f8c146bc0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c0b3cc0 con 0x7f3f8c146bc0 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c140cd0 con 0x7f3f8c0b38e0 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8c0b3e40 con 0x7f3f8c140890 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.778+0000 7f3f90e84640 10 monclient(hunting): authenticate will time out at 1310.968722s 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c146bc0 0x7f3f8c1d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f89d9b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c140890 0x7f3f8c0b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c146bc0 0x7f3f8c1d7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41962/0 (socket says 172.21.15.83:41962) 2024-03-25T01:59:51.786 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:43246/0 (socket says 172.21.15.83:43246) 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8ad9d640 1 -- 172.21.15.83:0/695179165 learned_addr learned my addr 172.21.15.83:0/695179165 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8ad9d640 10 monclient(hunting): get_auth_request con 0x7f3f8c146bc0 auth_method 0 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8a59c640 10 monclient(hunting): get_auth_request con 0x7f3f8c0b38e0 auth_method 0 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.779+0000 7f3f8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f89d9b640 10 monclient(hunting): get_auth_request con 0x7f3f8c140890 auth_method 0 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:51.787 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:51.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:51.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:51.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:51.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:51.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:51.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:51.788 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.780+0000 7f3f89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:51.789 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 1 -- 172.21.15.83:0/695179165 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c140890 msgr2=0x7f3f8c0b8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c140890 0x7f3f8c0b8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 1 -- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f8c0b38e0 msgr2=0x7f3f8c0b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 10 monclient: _finish_hunting 0 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 1 monclient: found mon.noname-a 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.783+0000 7f3f8ad9d640 1 -- 172.21.15.83:0/695179165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c0b8cb0 con 0x7f3f8c146bc0 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.786+0000 7f3f8ad9d640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c146bc0 0x7f3f8c1d7f60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400ca20 tx=0x7f3f8400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.786+0000 7f3f7affd640 10 client.?.objecter ms_handle_connect 0x7f3f8c146bc0 2024-03-25T01:59:51.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.786+0000 7f3f7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:51.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 1 -- 172.21.15.83:0/695179165 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84017070 con 0x7f3f8c146bc0 2024-03-25T01:59:51.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:51.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T01:59:51.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: dump: 2024-03-25T01:59:51.795 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:51.795 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: _finish_auth 0 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: _check_auth_tickets 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 1 -- 172.21.15.83:0/695179165 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f8400abd0 con 0x7f3f8c146bc0 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:51.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 1 -- 172.21.15.83:0/695179165 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84012400 con 0x7f3f8c146bc0 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f7affd640 10 monclient: dump: 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f88d99640 4 set_mon_vals no callback set 2024-03-25T01:59:51.797 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f90e84640 5 monclient: authenticate success, global_id 4218 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f90e84640 10 monclient: _renew_subs 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f90e84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.791+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8c0b8f60 con 0x7f3f8c146bc0 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.792+0000 7f3f90e84640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.792+0000 7f3f90e84640 10 monclient: _renew_subs 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.792+0000 7f3f90e84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.792+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8c143600 con 0x7f3f8c146bc0 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.792+0000 7f3f90e84640 1 librados: init done 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.793+0000 7f3f90e84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.793+0000 7f3f90e84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.793+0000 7f3f90e84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:51.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.793+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f8c04f940 con 0x7f3f8c146bc0 2024-03-25T01:59:51.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.822+0000 7f3f7affd640 1 -- 172.21.15.83:0/695179165 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7f3f8401a070 con 0x7f3f8c146bc0 2024-03-25T01:59:51.824 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.823+0000 7f3f7affd640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f6c06da10 0x7f3f6c06fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:51.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.823+0000 7f3f8a59c640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f6c06da10 0x7f3f6c06fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:51.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.823+0000 7f3f8a59c640 10 monclient: get_auth_request con 0x7f3f6c06da10 auth_method 0 2024-03-25T01:59:51.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.824+0000 7f3f7affd640 10 client.4218.objecter ms_handle_connect 0x7f3f6c06da10 2024-03-25T01:59:51.825 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:51.824+0000 7f3f8a59c640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f6c06da10 0x7f3f6c06fed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c009b40 tx=0x7f3f7c009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:51.922 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:17.287 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 1 -- 172.21.15.83:0/695179165 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f3f84058060 con 0x7f3f8c146bc0 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 10 client.4218.objecter ms_dispatch 0x7f3f8c000b90 osd_map(10..10 src has 1..10) 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 1 -- 172.21.15.83:0/695179165 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3f84021020 con 0x7f3f8c146bc0 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:52.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.096+0000 7f3f7affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T01:59:52.116 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:17.481 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:52.157 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:17.520 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:52.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.405+0000 7f3f90e84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:52.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.405+0000 7f3f90e84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:52.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.405+0000 7f3f90e84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T01:59:52.407 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.405+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f8c0b9130 con 0x7f3f8c146bc0 2024-03-25T01:59:52.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.432+0000 7f3f7affd640 1 -- 172.21.15.83:0/695179165 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f3f8c143600 con 0x7f3f8c146bc0 2024-03-25T01:59:52.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.432+0000 7f3f7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T01:59:52.434 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.432+0000 7f3f7affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T01:59:52.434 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T01:59:52.434 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T01:59:34.053585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 10 librados: watch_flush enter 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 10 librados: watch_flush exit 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 10 client.4218.objecter successfully canceled tick 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f6c06da10 msgr2=0x7f3f6c06fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f6c06da10 0x7f3f6c06fed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c009b40 tx=0x7f3f7c009190 comp rx=0 tx=0).stop 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 10 monclient: shutdown 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c146bc0 msgr2=0x7f3f8c1d7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c146bc0 0x7f3f8c1d7f60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400ca20 tx=0x7f3f8400cee0 comp rx=0 tx=0).stop 2024-03-25T01:59:52.438 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 shutdown_connections 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f6c06da10 0x7f3f6c06fed0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f8c146bc0 0x7f3f8c1d7f60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f8c140890 0x7f3f8c0b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 --2- 172.21.15.83:0/695179165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.434+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 >> 172.21.15.83:0/695179165 conn(0x7f3f8c067460 msgr2=0x7f3f8c06a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.436+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 shutdown_connections 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.436+0000 7f3f90e84640 1 -- 172.21.15.83:0/695179165 wait complete. 2024-03-25T01:59:52.439 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:52.436+0000 7f3f90e84640 1 librados: shutdown 2024-03-25T01:59:52.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T01:59:55.895 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:21.271 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:55.899 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:21.275 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.087 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:21.462 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.093 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:21.468 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.120 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:21.496 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.124 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:21.500 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.195 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:21.570 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.201 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:21.576 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.757 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:22.116 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:56.761 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:22.120 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:57.229 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:22.594 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:57.235 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:22.600 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:57.393 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:22.758 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:57.397 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:22.762 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:57.412 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:22.775 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:57.416 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:22.779 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T01:59:59.254 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.364+0000 7fd6dd486640 10 monclient: get_monmap_and_config 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.364+0000 7fd6dd486640 10 monclient: build_initial_monmap 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.364+0000 7fd6dd486640 10 monclient: monmap: 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:59:59.365161+0000 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:59:59.365161+0000 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:59.366 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.365+0000 7fd6dd486640 1 Processor -- start 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.365+0000 7fd6dd486640 1 -- start start 2024-03-25T01:59:59.367 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.365+0000 7fd6dd486640 10 monclient: init 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.365+0000 7fd6dd486640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.365+0000 7fd6dd486640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.365+0000 7fd6dd486640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd6d80b5a10 0x7fd6d8143750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.365+0000 7fd6dd486640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d80b5a10 addr v2:172.21.15.83:3300/0 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6d8143c90 addr v1:172.21.15.153:6789/0 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8147330 0x7fd6d814da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d8147330 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d8144100 con 0x7fd6d8143c90 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d80b5df0 con 0x7fd6d80b5a10 2024-03-25T01:59:59.371 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d80ab9c0 con 0x7fd6d8147330 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6dd486640 10 monclient(hunting): authenticate will time out at 1318.555093s 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6d77fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8147330 0x7fd6d814da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.366+0000 7fd6d6ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd6d80b5a10 0x7fd6d8143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8147330 0x7fd6d814da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:52224/0 (socket says 172.21.15.83:52224) 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 1 -- 172.21.15.83:0/3570443861 learned_addr learned my addr 172.21.15.83:0/3570443861 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 10 monclient(hunting): get_auth_request con 0x7fd6d8147330 auth_method 0 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d6ffd640 10 monclient(hunting): get_auth_request con 0x7fd6d80b5a10 auth_method 0 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:59.372 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.367+0000 7fd6d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.368+0000 7fd6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.368+0000 7fd6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.368+0000 7fd6d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 1 -- 172.21.15.83:0/3570443861 >> v2:172.21.15.83:3300/0 conn(0x7fd6d80b5a10 msgr2=0x7fd6d8143750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 1 --2- 172.21.15.83:0/3570443861 >> v2:172.21.15.83:3300/0 conn(0x7fd6d80b5a10 0x7fd6d8143750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 1 -- 172.21.15.83:0/3570443861 >> v1:172.21.15.153:6789/0 conn(0x7fd6d8143c90 legacy=0x7fd6d8146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 10 monclient: _finish_hunting 0 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 1 monclient: found mon.noname-b 2024-03-25T01:59:59.373 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:59.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T01:59:59.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.369+0000 7fd6d77fe640 1 -- 172.21.15.83:0/3570443861 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d80abbf0 con 0x7fd6d8147330 2024-03-25T01:59:59.374 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.372+0000 7fd6d77fe640 1 --2- 172.21.15.83:0/3570443861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8147330 0x7fd6d814da90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009e50 tx=0x7fd6d002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d64914b5d6bc8ae7 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.376+0000 7fd6d5ffb640 1 -- 172.21.15.83:0/3570443861 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d002cb00 con 0x7fd6d8147330 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: dump: 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.380 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: _finish_auth 0 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6dd486640 5 monclient: authenticate success, global_id 4169 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6dd486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 1 -- 172.21.15.83:0/3570443861 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6d002cca0 con 0x7fd6d8147330 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:59.381 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6dd486640 10 monclient: get_monmap_and_config success 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6dd486640 4 set_mon_vals no callback set 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 1 -- 172.21.15.83:0/3570443861 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0034610 con 0x7fd6d8147330 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.377+0000 7fd6d5ffb640 10 monclient: dump: 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:59.382 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.378+0000 7fd6dd486640 10 monclient: shutdown 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.378+0000 7fd6dd486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.378+0000 7fd6dd486640 1 -- 172.21.15.83:0/3570443861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8147330 msgr2=0x7fd6d814da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.378+0000 7fd6dd486640 1 --2- 172.21.15.83:0/3570443861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8147330 0x7fd6d814da90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009e50 tx=0x7fd6d002bf60 comp rx=0 tx=0).stop 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.379+0000 7fd6dd486640 1 -- 172.21.15.83:0/3570443861 shutdown_connections 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.379+0000 7fd6dd486640 1 --2- 172.21.15.83:0/3570443861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8147330 0x7fd6d814da90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.379+0000 7fd6dd486640 1 --2- 172.21.15.83:0/3570443861 >> v2:172.21.15.83:3300/0 conn(0x7fd6d80b5a10 0x7fd6d8143750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.379+0000 7fd6dd486640 1 -- 172.21.15.83:0/3570443861 >> 172.21.15.83:0/3570443861 conn(0x7fd6d8067460 msgr2=0x7fd6d8067870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.379+0000 7fd6dd486640 1 -- 172.21.15.83:0/3570443861 shutdown_connections 2024-03-25T01:59:59.383 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.379+0000 7fd6dd486640 1 -- 172.21.15.83:0/3570443861 wait complete. 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 10 monclient: build_initial_monmap 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 10 monclient: monmap: 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T01:59:59.390 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 1 librados: starting msgr at 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 1 librados: starting objecter 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 1 Processor -- start 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 1 -- start start 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 1 librados: setting wanted keys 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 1 librados: calling monclient init 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.380+0000 7fd6dd486640 10 monclient: init 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.381+0000 7fd6dd486640 10 monclient: _reopen_session rank -1 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.381+0000 7fd6dd486640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T01:59:59.391 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.381+0000 7fd6dd486640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d80b5a10 0x7fd6d80accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.381+0000 7fd6dd486640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d80b5a10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.381+0000 7fd6dd486640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8143c90 0x7fd6d806f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.381+0000 7fd6dd486640 10 monclient(hunting): picked mon.noname-c con 0x7fd6d8143c90 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.381+0000 7fd6dd486640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6d8147330 0x7fd6d806fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d8147330 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d8143180 con 0x7fd6d8147330 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d8146790 con 0x7fd6d80b5a10 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 10 monclient(hunting): start opening mon connection 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d80aaf00 con 0x7fd6d8143c90 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 10 monclient(hunting): _renew_subs 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6dd486640 10 monclient(hunting): authenticate will time out at 1318.571187s 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d80b5a10 0x7fd6d80accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:59.392 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d67fc640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8143c90 0x7fd6d806f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6d8147330 0x7fd6d806fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d6ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d80b5a10 0x7fd6d80accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:58952/0 (socket says 172.21.15.83:58952) 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d6ffd640 1 -- 172.21.15.83:0/4038339302 learned_addr learned my addr 172.21.15.83:0/4038339302 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d6ffd640 10 monclient(hunting): get_auth_request con 0x7fd6d80b5a10 auth_method 0 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.382+0000 7fd6d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d77fe640 10 monclient(hunting): get_auth_request con 0x7fd6d8147330 auth_method 0 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d67fc640 10 monclient(hunting): get_auth_request con 0x7fd6d8143c90 auth_method 0 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T01:59:59.393 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.383+0000 7fd6d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 10 monclient(hunting): handle_auth_done global_id 4222 payload 824 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 1 -- 172.21.15.83:0/4038339302 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8143c90 msgr2=0x7fd6d806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:59.394 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8143c90 0x7fd6d806f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 1 -- 172.21.15.83:0/4038339302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6d8147330 msgr2=0x7fd6d806fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6d8147330 0x7fd6d806fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 1 monclient: found mon.noname-b 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.385+0000 7fd6d6ffd640 1 -- 172.21.15.83:0/4038339302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d80ad880 con 0x7fd6d80b5a10 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.388+0000 7fd6d6ffd640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d80b5a10 0x7fd6d80accb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800e9e0 tx=0x7fd6c800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.388+0000 7fd6bf7fe640 10 client.?.objecter ms_handle_connect 0x7fd6d80b5a10 2024-03-25T01:59:59.395 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.388+0000 7fd6bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 1 -- 172.21.15.83:0/4038339302 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8017070 con 0x7fd6d80b5a10 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: dump: 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.397 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: _finish_auth 0 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 1 -- 172.21.15.83:0/4038339302 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c800c9d0 con 0x7fd6d80b5a10 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 1 -- 172.21.15.83:0/4038339302 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800cdc0 con 0x7fd6d80b5a10 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T01:59:59.398 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6bf7fe640 10 monclient: dump: 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 5 monclient: authenticate success, global_id 4222 2024-03-25T01:59:59.399 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6d57fa640 4 set_mon_vals no callback set 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 10 monclient: _renew_subs 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d80ad1f0 con 0x7fd6d80b5a10 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 10 monclient: _renew_subs 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d80adda0 con 0x7fd6d80b5a10 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.393+0000 7fd6dd486640 1 librados: init done 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.394+0000 7fd6dd486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.394+0000 7fd6dd486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.394+0000 7fd6dd486640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T01:59:59.400 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.394+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d813f140 con 0x7fd6d80b5a10 2024-03-25T01:59:59.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.426+0000 7fd6bf7fe640 1 -- 172.21.15.83:0/4038339302 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7fd6c801a070 con 0x7fd6d80b5a10 2024-03-25T01:59:59.427 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.426+0000 7fd6bf7fe640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd6c406d9e0 0x7fd6c406fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T01:59:59.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.427+0000 7fd6d67fc640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd6c406d9e0 0x7fd6c406fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T01:59:59.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.427+0000 7fd6d67fc640 10 monclient: get_auth_request con 0x7fd6c406d9e0 auth_method 0 2024-03-25T01:59:59.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.427+0000 7fd6bf7fe640 10 client.4222.objecter ms_handle_connect 0x7fd6c406d9e0 2024-03-25T01:59:59.428 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.427+0000 7fd6d67fc640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd6c406d9e0 0x7fd6c406fea0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8075af0 tx=0x7fd6cc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T01:59:59.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.431+0000 7fd6bf7fe640 1 -- 172.21.15.83:0/4038339302 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fd6c8058730 con 0x7fd6d80b5a10 2024-03-25T01:59:59.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.431+0000 7fd6bf7fe640 10 client.4222.objecter ms_dispatch 0x7fd6d8000b90 osd_map(10..10 src has 1..10) 2024-03-25T01:59:59.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.431+0000 7fd6bf7fe640 3 client.4222.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T01:59:59.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.431+0000 7fd6bf7fe640 3 client.4222.objecter handle_osd_map decoding full epoch 10 2024-03-25T01:59:59.432 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.431+0000 7fd6bf7fe640 20 client.4222.objecter dump_active .. 0 homeless 2024-03-25T01:59:59.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.797+0000 7fd6bf7fe640 1 -- 172.21.15.83:0/4038339302 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd6c8059aa0 con 0x7fd6d80b5a10 2024-03-25T01:59:59.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.797+0000 7fd6bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T01:59:59.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T01:59:59.797+0000 7fd6bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:00.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.097+0000 7fd6dd486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:00.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.097+0000 7fd6dd486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:00.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.097+0000 7fd6dd486640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:00.098 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.097+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6d80692f0 con 0x7fd6d80b5a10 2024-03-25T02:00:00.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.165+0000 7fd6bf7fe640 1 -- 172.21.15.83:0/4038339302 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fd6c805db40 con 0x7fd6d80b5a10 2024-03-25T02:00:00.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.165+0000 7fd6bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:00.166 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.165+0000 7fd6bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:00.166 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T02:00:00.166 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T01:59:34.053585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:00:00.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 10 librados: watch_flush enter 2024-03-25T02:00:00.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 10 librados: watch_flush exit 2024-03-25T02:00:00.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 20 client.4222.objecter shutdown clearing up homeless session... 2024-03-25T02:00:00.169 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 10 client.4222.objecter successfully canceled tick 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd6c406d9e0 msgr2=0x7fd6c406fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd6c406d9e0 0x7fd6c406fea0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8075af0 tx=0x7fd6cc009040 comp rx=0 tx=0).stop 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 10 monclient: shutdown 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d80b5a10 msgr2=0x7fd6d80accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d80b5a10 0x7fd6d80accb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800e9e0 tx=0x7fd6c800eea0 comp rx=0 tx=0).stop 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 shutdown_connections 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd6c406d9e0 0x7fd6c406fea0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6d8147330 0x7fd6d806fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd6d8143c90 0x7fd6d806f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 --2- 172.21.15.83:0/4038339302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d80b5a10 0x7fd6d80accb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 >> 172.21.15.83:0/4038339302 conn(0x7fd6d8067460 msgr2=0x7fd6d8145b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:00.170 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 shutdown_connections 2024-03-25T02:00:00.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 -- 172.21.15.83:0/4038339302 wait complete. 2024-03-25T02:00:00.171 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:00.168+0000 7fd6dd486640 1 librados: shutdown 2024-03-25T02:00:00.182 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T02:00:00.224 INFO:tasks.ceph.osd.5.smithi153.stderr:==00:00:00:25.600 35365== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:00.231 INFO:tasks.ceph.osd.7.smithi153.stderr:==00:00:00:25.606 35367== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:00.371 INFO:tasks.ceph.osd.4.smithi153.stderr:==00:00:00:25.747 35364== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:00.513 INFO:tasks.ceph.osd.6.smithi153.stderr:==00:00:00:25.888 35366== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:01.131 INFO:tasks.ceph.osd.3.smithi083.stderr:==00:00:00:26.490 35390== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:01.533 INFO:tasks.ceph.osd.0.smithi083.stderr:==00:00:00:26.898 35387== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:01.902 INFO:tasks.ceph.osd.1.smithi083.stderr:==00:00:00:27.267 35388== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:01.978 INFO:tasks.ceph.osd.2.smithi083.stderr:==00:00:00:27.341 35389== Warning: unimplemented fcntl command: 1038 2024-03-25T02:00:06.994 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T02:00:07.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.119+0000 7fe8d2395640 10 monclient: get_monmap_and_config 2024-03-25T02:00:07.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.119+0000 7fe8d2395640 10 monclient: build_initial_monmap 2024-03-25T02:00:07.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.119+0000 7fe8d2395640 10 monclient: monmap: 2024-03-25T02:00:07.120 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:07.120 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:07.120 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T02:00:07.119949+0000 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T02:00:07.119949+0000 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:07.121 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.120+0000 7fe8d2395640 1 Processor -- start 2024-03-25T02:00:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.120+0000 7fe8d2395640 1 -- start start 2024-03-25T02:00:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.120+0000 7fe8d2395640 10 monclient: init 2024-03-25T02:00:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.120+0000 7fe8d2395640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.120+0000 7fe8d2395640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-03-25T02:00:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.120+0000 7fe8d2395640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8cc0b38e0 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:07.123 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.120+0000 7fe8d2395640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe8cc140a60 0x7fe8cc146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 10 monclient(hunting): picked mon.noname-a con 0x7fe8cc140a60 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8cc146bc0 0x7fe8cc14da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 10 monclient(hunting): picked mon.noname-c con 0x7fe8cc146bc0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8cc0b3300 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe8cc0ab060 con 0x7fe8cc140a60 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fe8cc0abed0 con 0x7fe8cc146bc0 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d2395640 10 monclient(hunting): authenticate will time out at 1326.310022s 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8cb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe8cc140a60 0x7fe8cc146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.121+0000 7fe8d0932640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8cc146bc0 0x7fe8cc14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8cb7fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe8cc140a60 0x7fe8cc146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43590/0 (socket says 172.21.15.83:43590) 2024-03-25T02:00:07.124 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8cb7fe640 1 -- 172.21.15.83:0/1155296353 learned_addr learned my addr 172.21.15.83:0/1155296353 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8cb7fe640 10 monclient(hunting): get_auth_request con 0x7fe8cc140a60 auth_method 0 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8d0932640 10 monclient(hunting): get_auth_request con 0x7fe8cc146bc0 auth_method 0 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8d0932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8d0932640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:07.125 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.122+0000 7fe8d0932640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:07.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.123+0000 7fe8d0932640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:07.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.123+0000 7fe8d0932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:07.126 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.123+0000 7fe8d0932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.123+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.123+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.123+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 10 monclient(hunting): handle_auth_done global_id 4225 payload 274 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 1 -- 172.21.15.83:0/1155296353 >> v2:172.21.15.83:3300/0 conn(0x7fe8cc140a60 msgr2=0x7fe8cc146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 1 --2- 172.21.15.83:0/1155296353 >> v2:172.21.15.83:3300/0 conn(0x7fe8cc140a60 0x7fe8cc146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 1 -- 172.21.15.83:0/1155296353 >> v1:172.21.15.83:6789/0 conn(0x7fe8cc0b38e0 legacy=0x7fe8cc140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 10 monclient: _finish_hunting 0 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 1 monclient: found mon.noname-c 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:07.127 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.125+0000 7fe8d0932640 1 -- 172.21.15.83:0/1155296353 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8cc0ac1b0 con 0x7fe8cc146bc0 2024-03-25T02:00:07.129 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.128+0000 7fe8d0932640 1 --2- 172.21.15.83:0/1155296353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc146bc0 0x7fe8cc14da10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc140ea0 tx=0x7fe8c402bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf0e6b7b9b2e571 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:07.136 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 1 -- 172.21.15.83:0/1155296353 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4039070 con 0x7fe8cc146bc0 2024-03-25T02:00:07.136 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:07.136 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:07.136 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: dump: 2024-03-25T02:00:07.136 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:07.136 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:07.136 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 1 -- 172.21.15.83:0/1155296353 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8c402cc70 con 0x7fe8cc146bc0 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8d2395640 5 monclient: authenticate success, global_id 4225 2024-03-25T02:00:07.137 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8d2395640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8d2395640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8d2395640 4 set_mon_vals no callback set 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 1 -- 172.21.15.83:0/1155296353 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4034660 con 0x7fe8cc146bc0 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.133+0000 7fe8caffd640 10 monclient: dump: 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.138 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 10 monclient: shutdown 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 -- 172.21.15.83:0/1155296353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc146bc0 msgr2=0x7fe8cc14da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1155296353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc146bc0 0x7fe8cc14da10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc140ea0 tx=0x7fe8c402bec0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 -- 172.21.15.83:0/1155296353 shutdown_connections 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1155296353 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc146bc0 0x7fe8cc14da10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1155296353 >> v2:172.21.15.83:3300/0 conn(0x7fe8cc140a60 0x7fe8cc146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.139 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 -- 172.21.15.83:0/1155296353 >> 172.21.15.83:0/1155296353 conn(0x7fe8cc067460 msgr2=0x7fe8cc067870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 -- 172.21.15.83:0/1155296353 shutdown_connections 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.134+0000 7fe8d2395640 1 -- 172.21.15.83:0/1155296353 wait complete. 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 10 monclient: build_initial_monmap 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 10 monclient: monmap: 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:07.140 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:07.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 1 librados: starting msgr at 2024-03-25T02:00:07.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 1 librados: starting objecter 2024-03-25T02:00:07.141 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 1 Processor -- start 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 1 -- start start 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 1 librados: setting wanted keys 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 1 librados: calling monclient init 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.135+0000 7fe8d2395640 10 monclient: init 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8cc0b38e0 0x7fe8cc145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:07.146 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): picked mon.noname-a con 0x7fe8cc0b38e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc140a60 0x7fe8cc1d8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): picked mon.noname-b con 0x7fe8cc140a60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe8cc146bc0 0x7fe8cc1d8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): picked mon.noname-c con 0x7fe8cc146bc0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe8cc0b3e40 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe8cc0b3cc0 con 0x7fe8cc140a60 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fe8cc147010 con 0x7fe8cc146bc0 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:07.147 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.136+0000 7fe8d2395640 10 monclient(hunting): authenticate will time out at 1326.325560s 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc140a60 0x7fe8cc1d8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8cc0b38e0 0x7fe8cc145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8d0932640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe8cc146bc0 0x7fe8cc1d8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8cc0b38e0 0x7fe8cc145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43600/0 (socket says 172.21.15.83:43600) 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cbfff640 1 -- 172.21.15.83:0/1085939342 learned_addr learned my addr 172.21.15.83:0/1085939342 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request con 0x7fe8cc0b38e0 auth_method 0 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cb7fe640 10 monclient(hunting): get_auth_request con 0x7fe8cc140a60 auth_method 0 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:07.148 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8d0932640 10 monclient(hunting): get_auth_request con 0x7fe8cc146bc0 auth_method 0 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8d0932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8d0932640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.137+0000 7fe8d0932640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8d0932640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8d0932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:07.149 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.138+0000 7fe8d0932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 1 -- 172.21.15.83:0/1085939342 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe8cc146bc0 msgr2=0x7fe8cc1d8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe8cc146bc0 0x7fe8cc1d8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 1 -- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc140a60 msgr2=0x7fe8cc1d8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc140a60 0x7fe8cc1d8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 1 monclient: found mon.noname-a 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.141+0000 7fe8cbfff640 1 -- 172.21.15.83:0/1085939342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8cc1424f0 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.144+0000 7fe8cbfff640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8cc0b38e0 0x7fe8cc145b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc00c430 tx=0x7fe8bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.144+0000 7fe8c8ff9640 10 client.?.objecter ms_handle_connect 0x7fe8cc0b38e0 2024-03-25T02:00:07.150 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.144+0000 7fe8c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:07.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 1 -- 172.21.15.83:0/1085939342 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc0052d0 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:07.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:07.152 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: dump: 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: _finish_auth 0 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:07.153 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 1 -- 172.21.15.83:0/1085939342 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8bc005490 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8d2395640 5 monclient: authenticate success, global_id 4230 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8d2395640 10 monclient: _renew_subs 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8d2395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8cc1465d0 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 1 -- 172.21.15.83:0/1085939342 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc007260 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8c8ff9640 10 monclient: dump: 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.154 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.149+0000 7fe8ca7fc640 4 set_mon_vals no callback set 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.150+0000 7fe8d2395640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.150+0000 7fe8d2395640 10 monclient: _renew_subs 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.150+0000 7fe8d2395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.150+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8cc146230 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.150+0000 7fe8d2395640 1 librados: init done 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.151+0000 7fe8d2395640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:07.155 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.151+0000 7fe8d2395640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:07.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.151+0000 7fe8d2395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:07.156 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.151+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8cc04f940 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.174+0000 7fe8c8ff9640 1 -- 172.21.15.83:0/1085939342 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7fe8bc007400 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.175 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.175+0000 7fe8c8ff9640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe8b806d9e0 0x7fe8b806fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:07.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.175+0000 7fe8cb7fe640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe8b806d9e0 0x7fe8b806fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:07.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.175+0000 7fe8cb7fe640 10 monclient: get_auth_request con 0x7fe8b806d9e0 auth_method 0 2024-03-25T02:00:07.176 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.176+0000 7fe8cb7fe640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe8b806d9e0 0x7fe8b806fea0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc1d5fd0 tx=0x7fe8c0009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:07.177 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.176+0000 7fe8c8ff9640 10 client.4230.objecter ms_handle_connect 0x7fe8b806d9e0 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.491+0000 7fe8c8ff9640 1 -- 172.21.15.83:0/1085939342 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe8bc057030 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.491+0000 7fe8c8ff9640 10 client.4230.objecter ms_dispatch 0x7fe8cc000b90 osd_map(10..10 src has 1..10) 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.492+0000 7fe8c8ff9640 3 client.4230.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.492+0000 7fe8c8ff9640 3 client.4230.objecter handle_osd_map decoding full epoch 10 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.492+0000 7fe8c8ff9640 20 client.4230.objecter dump_active .. 0 homeless 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.492+0000 7fe8c8ff9640 1 -- 172.21.15.83:0/1085939342 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe8bc058f90 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.492+0000 7fe8c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:07.493 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.492+0000 7fe8c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:07.740 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:07.736+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T02:00:07.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.763+0000 7fe8d2395640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:07.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.763+0000 7fe8d2395640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:07.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.763+0000 7fe8d2395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:07.764 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.763+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8cc1d8360 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.789+0000 7fe8c8ff9640 1 -- 172.21.15.83:0/1085939342 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe8cc1d8360 con 0x7fe8cc0b38e0 2024-03-25T02:00:07.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.789+0000 7fe8c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:07.790 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.789+0000 7fe8c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:07.790 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T02:00:07.790 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T01:59:34.053585+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:00:07.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 10 librados: watch_flush enter 2024-03-25T02:00:07.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 10 librados: watch_flush exit 2024-03-25T02:00:07.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 20 client.4230.objecter shutdown clearing up homeless session... 2024-03-25T02:00:07.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 10 client.4230.objecter successfully canceled tick 2024-03-25T02:00:07.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe8b806d9e0 msgr2=0x7fe8b806fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:07.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe8b806d9e0 0x7fe8b806fea0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc1d5fd0 tx=0x7fe8c0009240 comp rx=0 tx=0).stop 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 10 monclient: shutdown 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8cc0b38e0 msgr2=0x7fe8cc145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8cc0b38e0 0x7fe8cc145b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc00c430 tx=0x7fe8bc00c8f0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 shutdown_connections 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe8b806d9e0 0x7fe8b806fea0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe8cc146bc0 0x7fe8cc1d8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8cc140a60 0x7fe8cc1d8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 --2- 172.21.15.83:0/1085939342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8cc0b38e0 0x7fe8cc145b10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 >> 172.21.15.83:0/1085939342 conn(0x7fe8cc067460 msgr2=0x7fe8cc13b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 shutdown_connections 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 -- 172.21.15.83:0/1085939342 wait complete. 2024-03-25T02:00:07.793 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:07.791+0000 7fe8d2395640 1 librados: shutdown 2024-03-25T02:00:07.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T02:00:08.342 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T02:00:08.338+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T02:00:08.370 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:00:08.366+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T02:00:08.685 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:08.681+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T02:00:09.688 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:00:09.685+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T02:00:09.788 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:09.784+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T02:00:10.116 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:00:10.104+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T02:00:10.310 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:10.307+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T02:00:11.136 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T02:00:11.130+0000 bc8e640 -1 osd.4 0 waiting for initial osdmap 2024-03-25T02:00:11.580 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T02:00:11.565+0000 10255640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:13.346 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:00:13.342+0000 bc8e640 -1 osd.0 0 waiting for initial osdmap 2024-03-25T02:00:13.412 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:00:13.408+0000 bc8e640 -1 osd.3 0 waiting for initial osdmap 2024-03-25T02:00:13.826 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:00:13.805+0000 10255640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:13.844 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:00:13.829+0000 10255640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:14.613 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.705+0000 7f7cef8c8640 10 monclient: get_monmap_and_config 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.705+0000 7f7cef8c8640 10 monclient: build_initial_monmap 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.706+0000 7f7cef8c8640 10 monclient: monmap: 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T02:00:14.707411+0000 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T02:00:14.707411+0000 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:14.708 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:14.709 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:14.709 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:14.709 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:14.709 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:14.711 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.706+0000 7f7cef8c8640 1 Processor -- start 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.706+0000 7f7cef8c8640 1 -- start start 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.706+0000 7f7cef8c8640 10 monclient: init 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ce80b38e0 0x7f7ce8140350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f7ce80b38e0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ce8140890 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7ce8146d90 0x7f7ce814da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f7ce8146d90 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce80b3300 con 0x7f7ce8140890 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:14.712 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce80ab060 con 0x7f7ce80b38e0 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce80abed0 con 0x7f7ce8146d90 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.707+0000 7f7cef8c8640 10 monclient(hunting): authenticate will time out at 1333.897403s 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7ced664640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ce80b38e0 0x7f7ce8140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7cede65640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7ce8146d90 0x7f7ce814da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7ced664640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7ce80b38e0 0x7f7ce8140350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33462/0 (socket says 172.21.15.83:33462) 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7ced664640 1 -- 172.21.15.83:0/1655384902 learned_addr learned my addr 172.21.15.83:0/1655384902 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7ced664640 10 monclient(hunting): get_auth_request con 0x7f7ce80b38e0 auth_method 0 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7ced664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7ced664640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.708+0000 7f7ced664640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.709+0000 7f7ced664640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:14.713 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.709+0000 7f7ced664640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.709+0000 7f7ced664640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.709+0000 7f7cede65640 10 monclient(hunting): get_auth_request con 0x7f7ce8146d90 auth_method 0 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.709+0000 7f7cede65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.709+0000 7f7cede65640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.709+0000 7f7cede65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7cede65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7cede65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:14.714 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7cede65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 1 -- 172.21.15.83:0/1655384902 >> v2:172.21.15.153:3300/0 conn(0x7f7ce8146d90 msgr2=0x7f7ce814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 1 --2- 172.21.15.83:0/1655384902 >> v2:172.21.15.153:3300/0 conn(0x7f7ce8146d90 0x7f7ce814da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 1 -- 172.21.15.83:0/1655384902 >> v1:172.21.15.153:6789/0 conn(0x7f7ce8140890 legacy=0x7f7ce8146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 10 monclient: _finish_hunting 0 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 1 monclient: found mon.noname-a 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.710+0000 7f7ced664640 1 -- 172.21.15.83:0/1655384902 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce80ac1b0 con 0x7f7ce80b38e0 2024-03-25T02:00:14.715 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.713+0000 7f7ced664640 1 --2- 172.21.15.83:0/1655384902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce80b38e0 0x7f7ce8140350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80b5f10 tx=0x7f7ce402ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2935394525b18a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 1 -- 172.21.15.83:0/1655384902 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce402f870 con 0x7f7ce80b38e0 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: dump: 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.719 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: _finish_auth 0 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 1 -- 172.21.15.83:0/1655384902 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce402fa10 con 0x7f7ce80b38e0 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 1 -- 172.21.15.83:0/1655384902 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce402fe20 con 0x7f7ce80b38e0 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:14.720 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cdffff640 10 monclient: dump: 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cef8c8640 5 monclient: authenticate success, global_id 4254 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cef8c8640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:14.721 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.717+0000 7f7cef8c8640 4 set_mon_vals no callback set 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 10 monclient: shutdown 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1655384902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce80b38e0 msgr2=0x7f7ce8140350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1655384902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce80b38e0 0x7f7ce8140350 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80b5f10 tx=0x7f7ce402ba50 comp rx=0 tx=0).stop 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1655384902 shutdown_connections 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1655384902 >> v2:172.21.15.153:3300/0 conn(0x7f7ce8146d90 0x7f7ce814da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1655384902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce80b38e0 0x7f7ce8140350 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1655384902 >> 172.21.15.83:0/1655384902 conn(0x7f7ce8067460 msgr2=0x7f7ce8067870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1655384902 shutdown_connections 2024-03-25T02:00:14.725 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.718+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1655384902 wait complete. 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.719+0000 7f7cef8c8640 10 monclient: build_initial_monmap 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.719+0000 7f7cef8c8640 10 monclient: monmap: 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:14.726 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.720+0000 7f7cef8c8640 1 librados: starting msgr at 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.720+0000 7f7cef8c8640 1 librados: starting objecter 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.720+0000 7f7cef8c8640 1 Processor -- start 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.720+0000 7f7cef8c8640 1 -- start start 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.720+0000 7f7cef8c8640 1 librados: setting wanted keys 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.720+0000 7f7cef8c8640 1 librados: calling monclient init 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.720+0000 7f7cef8c8640 10 monclient: init 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7ce80b38e0 0x7f7ce81ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f7ce80b38e0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ce8140890 0x7f7ce81cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f7ce8140890 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:14.727 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce8146d90 0x7f7ce81cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f7ce8146d90 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce81471e0 con 0x7f7ce8146d90 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce8140cd0 con 0x7f7ce8140890 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce80b3cc0 con 0x7f7ce80b38e0 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7cef8c8640 10 monclient(hunting): authenticate will time out at 1333.911399s 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.721+0000 7f7ced664640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7ce80b38e0 0x7f7ce81ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7cede65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce8146d90 0x7f7ce81cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7ce80b38e0 0x7f7ce81ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54412/0 (socket says 172.21.15.83:54412) 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 1 -- 172.21.15.83:0/1404516578 learned_addr learned my addr 172.21.15.83:0/1404516578 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 10 monclient(hunting): get_auth_request con 0x7f7ce80b38e0 auth_method 0 2024-03-25T02:00:14.728 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7cede65640 10 monclient(hunting): get_auth_request con 0x7f7ce8146d90 auth_method 0 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7cede65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7cede65640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7cede65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7ced664640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.722+0000 7f7cece63640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ce8140890 0x7f7ce81cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cece63640 10 monclient(hunting): get_auth_request con 0x7f7ce8140890 auth_method 0 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cece63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cece63640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:14.729 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cece63640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:14.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cede65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:14.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cede65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:14.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cede65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:14.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cece63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:14.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cece63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:14.730 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.723+0000 7f7cece63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 1 -- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ce8140890 msgr2=0x7f7ce81cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ce8140890 0x7f7ce81cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 1 -- 172.21.15.83:0/1404516578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce8146d90 msgr2=0x7f7ce81cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce8146d90 0x7f7ce81cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 10 monclient: _finish_hunting 0 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 1 monclient: found mon.noname-c 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:14.731 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.725+0000 7f7ced664640 1 -- 172.21.15.83:0/1404516578 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce81cdfe0 con 0x7f7ce80b38e0 2024-03-25T02:00:14.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.728+0000 7f7ced664640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7ce80b38e0 0x7f7ce81ccdb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40090b0 tx=0x7f7ce4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:14.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.728+0000 7f7cddffb640 10 client.?.objecter ms_handle_connect 0x7f7ce80b38e0 2024-03-25T02:00:14.732 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.728+0000 7f7cddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 1 -- 172.21.15.83:0/1404516578 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4037bc0 con 0x7f7ce80b38e0 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: dump: 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.737 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 1 -- 172.21.15.83:0/1404516578 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce4046070 con 0x7f7ce80b38e0 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 1 -- 172.21.15.83:0/1404516578 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4041400 con 0x7f7ce80b38e0 2024-03-25T02:00:14.738 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cddffb640 10 monclient: dump: 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cef8c8640 5 monclient: authenticate success, global_id 4196 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cef8c8640 10 monclient: _renew_subs 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cef8c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:14.739 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce81ce970 con 0x7f7ce80b38e0 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.732+0000 7f7cdf7fe640 4 set_mon_vals no callback set 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.733+0000 7f7cef8c8640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.733+0000 7f7cef8c8640 10 monclient: _renew_subs 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.733+0000 7f7cef8c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.733+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce81cef30 con 0x7f7ce80b38e0 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.733+0000 7f7cef8c8640 1 librados: init done 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.734+0000 7f7cef8c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.734+0000 7f7cef8c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.734+0000 7f7cef8c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:14.740 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.734+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce804f940 con 0x7f7ce80b38e0 2024-03-25T02:00:14.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.764+0000 7f7cddffb640 1 -- 172.21.15.83:0/1404516578 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7f7ce4040070 con 0x7f7ce80b38e0 2024-03-25T02:00:14.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.764+0000 7f7cddffb640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f7ccc06d990 0x7f7ccc06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:14.766 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.764+0000 7f7cece63640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f7ccc06d990 0x7f7ccc06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:14.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.764+0000 7f7cece63640 10 monclient: get_auth_request con 0x7f7ccc06d990 auth_method 0 2024-03-25T02:00:14.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.765+0000 7f7cddffb640 10 client.4196.objecter ms_handle_connect 0x7f7ccc06d990 2024-03-25T02:00:14.767 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.765+0000 7f7cece63640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f7ccc06d990 0x7f7ccc06fe50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ce813a450 tx=0x7f7cd8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:14.770 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.768+0000 7f7cddffb640 1 -- 172.21.15.83:0/1404516578 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3071+0+0 (secure 0 0 0) 0x7f7ce407a4f0 con 0x7f7ce80b38e0 2024-03-25T02:00:14.770 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.768+0000 7f7cddffb640 10 client.4196.objecter ms_dispatch 0x7f7ce8000b90 osd_map(15..15 src has 1..15) 2024-03-25T02:00:14.770 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.768+0000 7f7cddffb640 3 client.4196.objecter handle_osd_map got epochs [15,15] > 0 2024-03-25T02:00:14.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.768+0000 7f7cddffb640 3 client.4196.objecter handle_osd_map decoding full epoch 15 2024-03-25T02:00:14.771 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:14.768+0000 7f7cddffb640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-25T02:00:15.091 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.090+0000 7f7cddffb640 1 -- 172.21.15.83:0/1404516578 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7ce4033030 con 0x7f7ce80b38e0 2024-03-25T02:00:15.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.090+0000 7f7cddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:15.092 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.090+0000 7f7cddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:15.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.275+0000 7f7cef8c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:15.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.275+0000 7f7cef8c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:15.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.275+0000 7f7cef8c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:15.277 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.275+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ce81cfc50 con 0x7f7ce80b38e0 2024-03-25T02:00:15.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.308+0000 7f7cddffb640 1 -- 172.21.15.83:0/1404516578 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8538 (secure 0 0 0) 0x7f7ce81cfc50 con 0x7f7ce80b38e0 2024-03-25T02:00:15.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.308+0000 7f7cddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:15.311 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.308+0000 7f7cddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:15.311 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T02:00:15.311 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":15,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T02:00:14.055213+0000","last_up_change":"2024-03-25T02:00:14.055213+0000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6801","nonce":2678950219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6803","nonce":2678950219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6807","nonce":2678950219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6805","nonce":2678950219}]},"public_addr":"172.21.15.83:6801/2678950219","cluster_addr":"172.21.15.83:6803/2678950219","heartbeat_back_addr":"172.21.15.83:6807/2678950219","heartbeat_front_addr":"172.21.15.83:6805/2678950219","state":["exists","up"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6809","nonce":3432501411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6811","nonce":3432501411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6815","nonce":3432501411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6813","nonce":3432501411}]},"public_addr":"172.21.15.83:6809/3432501411","cluster_addr":"172.21.15.83:6811/3432501411","heartbeat_back_addr":"172.21.15.83:6815/3432501411","heartbeat_front_addr":"172.21.15.83:6813/3432501411","state":["exists","up"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6819","nonce":2579144563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6821","nonce":2579144563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6825","nonce":2579144563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6823","nonce":2579144563}]},"public_addr":"172.21.15.153:6819/2579144563","cluster_addr":"172.21.15.153:6821/2579144563","heartbeat_back_addr":"172.21.15.153:6825/2579144563","heartbeat_front_addr":"172.21.15.153:6823/2579144563","state":["exists","up"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.336792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.311+0000 7f7cef8c8640 10 librados: watch_flush enter 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.311+0000 7f7cef8c8640 10 librados: watch_flush exit 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.311+0000 7f7cef8c8640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.311+0000 7f7cef8c8640 10 client.4196.objecter successfully canceled tick 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.311+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f7ccc06d990 msgr2=0x7f7ccc06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.311+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f7ccc06d990 0x7f7ccc06fe50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ce813a450 tx=0x7f7cd8009040 comp rx=0 tx=0).stop 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 10 monclient: shutdown 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:15.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7ce80b38e0 msgr2=0x7f7ce81ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7ce80b38e0 0x7f7ce81ccdb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7ce40090b0 tx=0x7f7ce4004730 comp rx=0 tx=0).stop 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 shutdown_connections 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f7ccc06d990 0x7f7ccc06fe50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ce8146d90 0x7f7ce81cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7ce8140890 0x7f7ce81cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 --2- 172.21.15.83:0/1404516578 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7ce80b38e0 0x7f7ce81ccdb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 >> 172.21.15.83:0/1404516578 conn(0x7f7ce8067460 msgr2=0x7f7ce8067840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 shutdown_connections 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 -- 172.21.15.83:0/1404516578 wait complete. 2024-03-25T02:00:15.315 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:15.312+0000 7f7cef8c8640 1 librados: shutdown 2024-03-25T02:00:15.326 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-03-25T02:00:18.226 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.055+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-25T02:00:18.278 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.225+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-25T02:00:18.326 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.277+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-03-25T02:00:18.327 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.299+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-25T02:00:18.392 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.326+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-25T02:00:18.419 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.379+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-03-25T02:00:18.442 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.419+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-03-25T02:00:18.442 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.440+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-25T02:00:18.491 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.471+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 3 2024-03-25T02:00:18.499 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.492+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-03-25T02:00:18.802 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.552+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-25T02:00:19.000 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:18.843+0000 5dc3480 -1 bdev(0x17e7b640 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-03-25T02:00:19.090 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.078+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-03-25T02:00:19.274 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.095+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.284 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.273+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.580 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.283+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-03-25T02:00:19.600 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.318+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.600 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:00:19.106+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-03-25T02:00:19.601 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.568+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.785 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.579+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.785 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.600+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.785 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.772+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.834 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:00:19.218+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-03-25T02:00:19.836 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:00:19.600+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-03-25T02:00:19.959 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:19.789+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-03-25T02:00:19.971 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:19.967+0000 b47a640 -1 osd.5 0 waiting for initial osdmap 2024-03-25T02:00:20.342 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:20.328+0000 5dc3480 -1 bdev(0x17d5d600 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-03-25T02:00:20.362 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:20.341+0000 5dc3480 -1 bdev(0x17d5d600 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-03-25T02:00:20.366 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:20.363+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-25T02:00:20.476 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:00:20.461+0000 1425d640 -1 osd.5 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:20.788 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:20.361+0000 5dc3480 -1 bdev(0x17d5d600 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-03-25T02:00:20.870 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:20.427+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-25T02:00:20.922 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:20.870+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-25T02:00:20.924 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:20.411+0000 5dc3480 -1 bdev(0x17d5d600 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-03-25T02:00:20.952 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:20.920+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-25T02:00:20.996 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:20.950+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-03-25T02:00:20.997 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:20.982+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-25T02:00:21.021 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:21.014+0000 b47a640 -1 osd.6 0 waiting for initial osdmap 2024-03-25T02:00:21.057 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:00:21.053+0000 bc8e640 -1 osd.7 0 waiting for initial osdmap 2024-03-25T02:00:21.168 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:20.922+0000 5dc3480 -1 bdev(0x17d5d600 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-03-25T02:00:21.407 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:21.136+0000 5dc3480 -1 bdev(0x17e75d10 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-03-25T02:00:21.546 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:00:21.531+0000 1425d640 -1 osd.6 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:21.610 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:00:21.594+0000 1425d640 -1 osd.7 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:22.132 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T02:00:22.169 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:22.164+0000 bc8e640 -1 osd.1 0 waiting for initial osdmap 2024-03-25T02:00:22.224 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.222+0000 7f27ff6a1640 10 monclient: get_monmap_and_config 2024-03-25T02:00:22.224 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.222+0000 7f27ff6a1640 10 monclient: build_initial_monmap 2024-03-25T02:00:22.224 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.222+0000 7f27ff6a1640 10 monclient: monmap: 2024-03-25T02:00:22.224 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:22.224 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:22.224 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T02:00:22.224069+0000 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T02:00:22.224069+0000 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:22.225 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:22.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.223+0000 7f27ff6a1640 1 Processor -- start 2024-03-25T02:00:22.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.223+0000 7f27ff6a1640 1 -- start start 2024-03-25T02:00:22.226 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.223+0000 7f27ff6a1640 10 monclient: init 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.223+0000 7f27ff6a1640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f27f80b38e0 0x7f27f8140350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient(hunting): picked mon.noname-c con 0x7f27f80b38e0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27f8140890 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f814da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient(hunting): picked mon.noname-b con 0x7f27f8146d90 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f80b3300 con 0x7f27f8140890 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f27f80ab060 con 0x7f27f80b38e0 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f27f80abed0 con 0x7f27f8146d90 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:22.227 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.224+0000 7f27ff6a1640 10 monclient(hunting): authenticate will time out at 1341.414078s 2024-03-25T02:00:22.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.260+0000 7f27fd43d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f27f80b38e0 0x7f27f8140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:22.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.260+0000 7f27fcc3c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f27f8140890 0x7f27f8146680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:49156/0 (socket says 172.21.15.83:49156) 2024-03-25T02:00:22.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.260+0000 7f27fcc3c640 1 -- 172.21.15.83:0/2926814756 learned_addr learned my addr 172.21.15.83:0/2926814756 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:22.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.260+0000 7f27fd43d640 10 monclient(hunting): get_auth_request con 0x7f27f80b38e0 auth_method 0 2024-03-25T02:00:22.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.260+0000 7f27fd43d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:22.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.260+0000 7f27fd43d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:22.262 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.260+0000 7f27fd43d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.261+0000 7f27fd43d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.261+0000 7f27fd43d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:22.263 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.261+0000 7f27fd43d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:22.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.263+0000 7f27fd43d640 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-03-25T02:00:22.265 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 1 -- 172.21.15.83:0/2926814756 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 msgr2=0x7f27f814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:22.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 1 --2- 172.21.15.83:0/2926814756 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f814da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:22.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 1 -- 172.21.15.83:0/2926814756 >> v1:172.21.15.153:6789/0 conn(0x7f27f8140890 legacy=0x7f27f8146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:22.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 10 monclient: _finish_hunting 0 2024-03-25T02:00:22.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 1 monclient: found mon.noname-c 2024-03-25T02:00:22.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:22.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:22.266 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.264+0000 7f27fd43d640 1 -- 172.21.15.83:0/2926814756 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f80ac1b0 con 0x7f27f80b38e0 2024-03-25T02:00:22.268 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.267+0000 7f27fd43d640 1 --2- 172.21.15.83:0/2926814756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f8140350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27f40097a0 tx=0x7f27f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bd84e5e1ca31c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:22.281 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.279+0000 7f27effff640 1 -- 172.21.15.83:0/2926814756 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4035020 con 0x7f27f80b38e0 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: dump: 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:22.282 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: _finish_auth 0 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 1 -- 172.21.15.83:0/2926814756 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f4031830 con 0x7f27f80b38e0 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 5 monclient: authenticate success, global_id 4261 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 4 set_mon_vals no callback set 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 1 -- 172.21.15.83:0/2926814756 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4031c00 con 0x7f27f80b38e0 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27effff640 10 monclient: dump: 2024-03-25T02:00:22.283 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:22.284 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:22.287 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 10 monclient: shutdown 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 -- 172.21.15.83:0/2926814756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 msgr2=0x7f27f8140350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/2926814756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f8140350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f27f40097a0 tx=0x7f27f402b8d0 comp rx=0 tx=0).stop 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 -- 172.21.15.83:0/2926814756 shutdown_connections 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/2926814756 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f814da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/2926814756 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f8140350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 -- 172.21.15.83:0/2926814756 >> 172.21.15.83:0/2926814756 conn(0x7f27f8067460 msgr2=0x7f27f8067870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 -- 172.21.15.83:0/2926814756 shutdown_connections 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.280+0000 7f27ff6a1640 1 -- 172.21.15.83:0/2926814756 wait complete. 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 10 monclient: build_initial_monmap 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 10 monclient: monmap: 2024-03-25T02:00:22.288 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 1 librados: starting msgr at 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 1 librados: starting objecter 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 1 Processor -- start 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 1 -- start start 2024-03-25T02:00:22.289 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 1 librados: setting wanted keys 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 1 librados: calling monclient init 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.281+0000 7f27ff6a1640 10 monclient: init 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): picked mon.noname-b con 0x7f27f80b38e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f8140890 0x7f27f80adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): picked mon.noname-a con 0x7f27f8140890 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f81d89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): picked mon.noname-c con 0x7f27f8146d90 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27f81471e0 con 0x7f27f8140890 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:22.290 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27f8140cd0 con 0x7f27f80b38e0 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f27f80b3cc0 con 0x7f27f8146d90 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.282+0000 7f27ff6a1640 10 monclient(hunting): authenticate will time out at 1341.472553s 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.283+0000 7f27fdc3e640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f81d89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.283+0000 7f27fdc3e640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f81d89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54428/0 (socket says 172.21.15.83:54428) 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.283+0000 7f27fdc3e640 1 -- 172.21.15.83:0/3104706440 learned_addr learned my addr 172.21.15.83:0/3104706440 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.283+0000 7f27fdc3e640 10 monclient(hunting): get_auth_request con 0x7f27f8146d90 auth_method 0 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.283+0000 7f27fdc3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.283+0000 7f27fdc3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.283+0000 7f27fdc3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fdc3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fdc3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:22.291 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fdc3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:22.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 10 monclient(hunting): get_auth_request con 0x7f27f80b38e0 auth_method 0 2024-03-25T02:00:22.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:22.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:22.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:22.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:22.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:22.292 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.284+0000 7f27fd43d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:22.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-03-25T02:00:22.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 1 -- 172.21.15.83:0/3104706440 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 msgr2=0x7f27f81d89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:22.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f81d89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:22.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 1 -- 172.21.15.83:0/3104706440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f8140890 msgr2=0x7f27f80adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:22.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f8140890 0x7f27f80adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:22.293 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 10 monclient: _finish_hunting 0 2024-03-25T02:00:22.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 1 monclient: found mon.noname-b 2024-03-25T02:00:22.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:22.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:22.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.287+0000 7f27fd43d640 1 -- 172.21.15.83:0/3104706440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f80af060 con 0x7f27f80b38e0 2024-03-25T02:00:22.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.290+0000 7f27fd43d640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f80ad350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27f4033000 tx=0x7f27f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:22.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.290+0000 7f27edffb640 10 client.?.objecter ms_handle_connect 0x7f27f80b38e0 2024-03-25T02:00:22.294 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.290+0000 7f27edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:22.296 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:22.290+0000 bc8e640 -1 osd.2 0 waiting for initial osdmap 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 1 -- 172.21.15.83:0/3104706440 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4041070 con 0x7f27f80b38e0 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: dump: 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.299 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 1 -- 172.21.15.83:0/3104706440 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f403aa30 con 0x7f27f80b38e0 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27ff6a1640 5 monclient: authenticate success, global_id 4264 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27ff6a1640 10 monclient: _renew_subs 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27ff6a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f80af340 con 0x7f27f80b38e0 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:22.300 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 1 -- 172.21.15.83:0/3104706440 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f4035040 con 0x7f27f80b38e0 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27edffb640 10 monclient: dump: 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.296+0000 7f27ef7fe640 4 set_mon_vals no callback set 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.297+0000 7f27ff6a1640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:22.301 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.297+0000 7f27ff6a1640 10 monclient: _renew_subs 2024-03-25T02:00:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.297+0000 7f27ff6a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.297+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f80b7cd0 con 0x7f27f80b38e0 2024-03-25T02:00:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.297+0000 7f27ff6a1640 1 librados: init done 2024-03-25T02:00:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.298+0000 7f27ff6a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.298+0000 7f27ff6a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:22.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.298+0000 7f27ff6a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:22.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.298+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f804f940 con 0x7f27f80b38e0 2024-03-25T02:00:22.322 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.321+0000 7f27edffb640 1 -- 172.21.15.83:0/3104706440 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7f27f404c060 con 0x7f27f80b38e0 2024-03-25T02:00:22.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.321+0000 7f27edffb640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f27dc06d9e0 0x7f27dc06fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:22.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.321+0000 7f27fcc3c640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f27dc06d9e0 0x7f27dc06fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:22.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.321+0000 7f27fcc3c640 10 monclient: get_auth_request con 0x7f27dc06d9e0 auth_method 0 2024-03-25T02:00:22.323 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.322+0000 7f27edffb640 10 client.4264.objecter ms_handle_connect 0x7f27dc06d9e0 2024-03-25T02:00:22.324 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.322+0000 7f27fcc3c640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f27dc06d9e0 0x7f27dc06fea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f27f80b9bf0 tx=0x7f27e8007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:22.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.566+0000 7f27edffb640 1 -- 172.21.15.83:0/3104706440 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3737+0+0 (secure 0 0 0) 0x7f27f403ae20 con 0x7f27f80b38e0 2024-03-25T02:00:22.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.566+0000 7f27edffb640 10 client.4264.objecter ms_dispatch 0x7f27f8000b90 osd_map(18..18 src has 1..18) 2024-03-25T02:00:22.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.566+0000 7f27edffb640 3 client.4264.objecter handle_osd_map got epochs [18,18] > 0 2024-03-25T02:00:22.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.566+0000 7f27edffb640 3 client.4264.objecter handle_osd_map decoding full epoch 18 2024-03-25T02:00:22.568 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.566+0000 7f27edffb640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T02:00:22.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.587+0000 7f27edffb640 1 -- 172.21.15.83:0/3104706440 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f27f407b700 con 0x7f27f80b38e0 2024-03-25T02:00:22.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.587+0000 7f27edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:22.589 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.587+0000 7f27edffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:22.702 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:00:22.686+0000 1425d640 -1 osd.1 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:22.761 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:00:22.738+0000 1425d640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-25T02:00:22.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.799+0000 7f27ff6a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:22.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.799+0000 7f27ff6a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:22.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.799+0000 7f27ff6a1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:22.801 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:22.799+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27f80b7f00 con 0x7f27f80b38e0 2024-03-25T02:00:23.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.115+0000 7f27edffb640 1 -- 172.21.15.83:0/3104706440 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+11893 (secure 0 0 0) 0x7f27f407b000 con 0x7f27f80b38e0 2024-03-25T02:00:23.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.115+0000 7f27edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:23.117 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.115+0000 7f27edffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:23.117 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T02:00:23.117 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":19,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T02:00:22.384492+0000","last_up_change":"2024-03-25T02:00:22.384492+0000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-25T02:00:14.575846+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6801","nonce":2678950219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6803","nonce":2678950219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6807","nonce":2678950219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6805","nonce":2678950219}]},"public_addr":"172.21.15.83:6801/2678950219","cluster_addr":"172.21.15.83:6803/2678950219","heartbeat_back_addr":"172.21.15.83:6807/2678950219","heartbeat_front_addr":"172.21.15.83:6805/2678950219","state":["exists","up"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6809","nonce":3432501411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6811","nonce":3432501411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6815","nonce":3432501411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6813","nonce":3432501411}]},"public_addr":"172.21.15.83:6809/3432501411","cluster_addr":"172.21.15.83:6811/3432501411","heartbeat_back_addr":"172.21.15.83:6815/3432501411","heartbeat_front_addr":"172.21.15.83:6813/3432501411","state":["exists","up"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6819","nonce":2579144563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6821","nonce":2579144563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6825","nonce":2579144563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6823","nonce":2579144563}]},"public_addr":"172.21.15.153:6819/2579144563","cluster_addr":"172.21.15.153:6821/2579144563","heartbeat_back_addr":"172.21.15.153:6825/2579144563","heartbeat_front_addr":"172.21.15.153:6823/2579144563","state":["exists","up"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6811","nonce":2458137144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6813","nonce":2458137144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6817","nonce":2458137144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6815","nonce":2458137144}]},"public_addr":"172.21.15.153:6811/2458137144","cluster_addr":"172.21.15.153:6813/2458137144","heartbeat_back_addr":"172.21.15.153:6817/2458137144","heartbeat_front_addr":"172.21.15.153:6815/2458137144","state":["exists","up"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6803","nonce":4205760170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6805","nonce":4205760170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6809","nonce":4205760170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6807","nonce":4205760170}]},"public_addr":"172.21.15.153:6803/4205760170","cluster_addr":"172.21.15.153:6805/4205760170","heartbeat_back_addr":"172.21.15.153:6809/4205760170","heartbeat_front_addr":"172.21.15.153:6807/4205760170","state":["exists","up"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6827","nonce":2580901001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6829","nonce":2580901001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6833","nonce":2580901001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6831","nonce":2580901001}]},"public_addr":"172.21.15.153:6827/2580901001","cluster_addr":"172.21.15.153:6829/2580901001","heartbeat_back_addr":"172.21.15.153:6833/2580901001","heartbeat_front_addr":"172.21.15.153:6831/2580901001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.217280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.630439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.336792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:00:23.119 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 10 librados: watch_flush enter 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 10 librados: watch_flush exit 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 10 client.4264.objecter successfully canceled tick 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f27dc06d9e0 msgr2=0x7f27dc06fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f27dc06d9e0 0x7f27dc06fea0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f27f80b9bf0 tx=0x7f27e8007a40 comp rx=0 tx=0).stop 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 10 monclient: shutdown 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 msgr2=0x7f27f80ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f80ad350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f27f4033000 tx=0x7f27f40046c0 comp rx=0 tx=0).stop 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 shutdown_connections 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f27dc06d9e0 0x7f27dc06fea0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27f8146d90 0x7f27f81d89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27f8140890 0x7f27f80adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:23.120 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 --2- 172.21.15.83:0/3104706440 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27f80b38e0 0x7f27f80ad350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:23.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 >> 172.21.15.83:0/3104706440 conn(0x7f27f8067460 msgr2=0x7f27f813ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:23.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 shutdown_connections 2024-03-25T02:00:23.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 -- 172.21.15.83:0/3104706440 wait complete. 2024-03-25T02:00:23.121 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:23.117+0000 7f27ff6a1640 1 librados: shutdown 2024-03-25T02:00:23.129 DEBUG:teuthology.misc:6 of 8 OSDs are up 2024-03-25T02:00:26.256 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:09.092 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.314 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:09.150 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.349 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:09.141 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.350 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:09.186 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.403 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:09.196 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.411 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:09.247 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.618 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:09.454 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.637 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:09.430 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:26.684 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:09.519 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:29.936 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T02:00:30.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.024+0000 7fbe96de9640 10 monclient: get_monmap_and_config 2024-03-25T02:00:30.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.024+0000 7fbe96de9640 10 monclient: build_initial_monmap 2024-03-25T02:00:30.027 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.024+0000 7fbe96de9640 10 monclient: monmap: 2024-03-25T02:00:30.027 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:30.027 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.027 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T02:00:30.026094+0000 2024-03-25T02:00:30.027 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T02:00:30.026094+0000 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.025+0000 7fbe96de9640 1 Processor -- start 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.025+0000 7fbe96de9640 1 -- start start 2024-03-25T02:00:30.028 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.025+0000 7fbe96de9640 10 monclient: init 2024-03-25T02:00:30.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.025+0000 7fbe96de9640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:30.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.025+0000 7fbe96de9640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-03-25T02:00:30.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.025+0000 7fbe96de9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbe900b38e0 0x7fbe90140350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 10 monclient(hunting): picked mon.noname-c con 0x7fbe900b38e0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:30.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe90140890 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:30.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe90146d90 0x7fbe9014da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.031 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 10 monclient(hunting): picked mon.noname-b con 0x7fbe90146d90 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe900b3300 con 0x7fbe90140890 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fbe900ab060 con 0x7fbe900b38e0 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fbe900abed0 con 0x7fbe90146d90 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe96de9640 10 monclient(hunting): authenticate will time out at 1349.216027s 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe95386640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe90146d90 0x7fbe9014da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.026+0000 7fbe94b85640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbe900b38e0 0x7fbe90140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe8ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbe90140890 0x7fbe90146680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:49526/0 (socket says 172.21.15.83:49526) 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe8ffff640 1 -- 172.21.15.83:0/162342354 learned_addr learned my addr 172.21.15.83:0/162342354 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:30.032 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe95386640 10 monclient(hunting): get_auth_request con 0x7fbe90146d90 auth_method 0 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe95386640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe95386640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe95386640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe94b85640 10 monclient(hunting): get_auth_request con 0x7fbe900b38e0 auth_method 0 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe94b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe94b85640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe94b85640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe95386640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe95386640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.027+0000 7fbe95386640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.028+0000 7fbe94b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.028+0000 7fbe94b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.028+0000 7fbe94b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.033 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 1 -- 172.21.15.83:0/162342354 >> v2:172.21.15.153:3300/0 conn(0x7fbe900b38e0 msgr2=0x7fbe90140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 1 --2- 172.21.15.83:0/162342354 >> v2:172.21.15.153:3300/0 conn(0x7fbe900b38e0 0x7fbe90140350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 1 -- 172.21.15.83:0/162342354 >> v1:172.21.15.153:6789/0 conn(0x7fbe90140890 legacy=0x7fbe90146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 10 monclient: _finish_hunting 0 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 1 monclient: found mon.noname-b 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.029+0000 7fbe95386640 1 -- 172.21.15.83:0/162342354 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe900ac1b0 con 0x7fbe90146d90 2024-03-25T02:00:30.034 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.031+0000 7fbe95386640 1 --2- 172.21.15.83:0/162342354 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe90146d90 0x7fbe9014da10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009d90 tx=0x7fbe8802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c870695676691498 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:30.035 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe8f7fe640 1 -- 172.21.15.83:0/162342354 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88039070 con 0x7fbe90146d90 2024-03-25T02:00:30.035 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe8f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe8f7fe640 10 monclient: dump: 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:30.036 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe8f7fe640 10 monclient: _finish_auth 0 2024-03-25T02:00:30.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe8f7fe640 10 monclient: _check_auth_tickets 2024-03-25T02:00:30.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:30.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe96de9640 5 monclient: authenticate success, global_id 4208 2024-03-25T02:00:30.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.033+0000 7fbe96de9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:30.037 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe8f7fe640 1 -- 172.21.15.83:0/162342354 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8802cc50 con 0x7fbe90146d90 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe8f7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe8f7fe640 1 -- 172.21.15.83:0/162342354 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88034440 con 0x7fbe90146d90 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 4 set_mon_vals no callback set 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe8f7fe640 10 monclient: dump: 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.038 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.039 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.039 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.039 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:30.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 10 monclient: shutdown 2024-03-25T02:00:30.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:30.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 -- 172.21.15.83:0/162342354 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe90146d90 msgr2=0x7fbe9014da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 --2- 172.21.15.83:0/162342354 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe90146d90 0x7fbe9014da10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009d90 tx=0x7fbe8802bea0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.042 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 -- 172.21.15.83:0/162342354 shutdown_connections 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 --2- 172.21.15.83:0/162342354 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe90146d90 0x7fbe9014da10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 --2- 172.21.15.83:0/162342354 >> v2:172.21.15.153:3300/0 conn(0x7fbe900b38e0 0x7fbe90140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 -- 172.21.15.83:0/162342354 >> 172.21.15.83:0/162342354 conn(0x7fbe90067460 msgr2=0x7fbe90067870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 -- 172.21.15.83:0/162342354 shutdown_connections 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.036+0000 7fbe96de9640 1 -- 172.21.15.83:0/162342354 wait complete. 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.037+0000 7fbe96de9640 10 monclient: build_initial_monmap 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.037+0000 7fbe96de9640 10 monclient: monmap: 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:epoch 0 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:created 0.000000 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.043 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.037+0000 7fbe96de9640 1 librados: starting msgr at 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.037+0000 7fbe96de9640 1 librados: starting objecter 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.037+0000 7fbe96de9640 1 Processor -- start 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 -- start start 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 librados: setting wanted keys 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 librados: calling monclient init 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient: init 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:00:30.044 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe900b38e0 0x7fbe900b7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): picked mon.noname-c con 0x7fbe900b38e0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe90140890 0x7fbe900b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): picked mon.noname-a con 0x7fbe90140890 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe90146d90 0x7fbe901d8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): picked mon.noname-b con 0x7fbe90146d90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901471e0 con 0x7fbe90140890 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90140cd0 con 0x7fbe90146d90 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fbe900b3cc0 con 0x7fbe900b38e0 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:30.045 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.038+0000 7fbe96de9640 10 monclient(hunting): authenticate will time out at 1349.228721s 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe94b85640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe900b38e0 0x7fbe900b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe90140890 0x7fbe900b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe95386640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe90146d90 0x7fbe901d8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe95386640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe90146d90 0x7fbe901d8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:54320/0 (socket says 172.21.15.83:54320) 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe95386640 1 -- 172.21.15.83:0/1980073417 learned_addr learned my addr 172.21.15.83:0/1980073417 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe95386640 10 monclient(hunting): get_auth_request con 0x7fbe90146d90 auth_method 0 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe95386640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe95386640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe95386640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe94b85640 10 monclient(hunting): get_auth_request con 0x7fbe900b38e0 auth_method 0 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe94b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe94b85640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.046 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.039+0000 7fbe94b85640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe8ffff640 10 monclient(hunting): get_auth_request con 0x7fbe90140890 auth_method 0 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe95386640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe95386640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe95386640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe94b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe94b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe94b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.047 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.040+0000 7fbe8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-03-25T02:00:30.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 1 -- 172.21.15.83:0/1980073417 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe900b38e0 msgr2=0x7fbe900b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.048 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe900b38e0 0x7fbe900b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 1 -- 172.21.15.83:0/1980073417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe90140890 msgr2=0x7fbe900b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe90140890 0x7fbe900b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 10 monclient: _finish_hunting 0 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 1 monclient: found mon.noname-b 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.042+0000 7fbe95386640 1 -- 172.21.15.83:0/1980073417 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe900b96a0 con 0x7fbe90146d90 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.045+0000 7fbe95386640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe90146d90 0x7fbe901d8610 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009eb0 tx=0x7fbe88004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.045+0000 7fbe8d7fa640 10 client.?.objecter ms_handle_connect 0x7fbe90146d90 2024-03-25T02:00:30.049 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.045+0000 7fbe8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:30.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.048+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88039050 con 0x7fbe90146d90 2024-03-25T02:00:30.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:30.051 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: dump: 2024-03-25T02:00:30.051 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: _finish_auth 0 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: _check_auth_tickets 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe88045070 con 0x7fbe90146d90 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 5 monclient: authenticate success, global_id 4276 2024-03-25T02:00:30.052 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 10 monclient: _renew_subs 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe901d80f0 con 0x7fbe90146d90 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88009450 con 0x7fbe90146d90 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8d7fa640 10 monclient: dump: 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:epoch 1 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.053 INFO:teuthology.misc.health.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr: 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe8effd640 4 set_mon_vals no callback set 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 10 monclient: _renew_subs 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe901d8b50 con 0x7fbe90146d90 2024-03-25T02:00:30.054 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.049+0000 7fbe96de9640 1 librados: init done 2024-03-25T02:00:30.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.050+0000 7fbe96de9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:30.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.050+0000 7fbe96de9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:30.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.050+0000 7fbe96de9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.055 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.050+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9004f940 con 0x7fbe90146d90 2024-03-25T02:00:30.302 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.300+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) ==== 90456+0+0 (secure 0 0 0) 0x7fbe8803e610 con 0x7fbe90146d90 2024-03-25T02:00:30.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.300+0000 7fbe8d7fa640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbe7406d990 0x7fbe7406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.301+0000 7fbe94b85640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbe7406d990 0x7fbe7406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.301+0000 7fbe94b85640 10 monclient: get_auth_request con 0x7fbe7406d990 auth_method 0 2024-03-25T02:00:30.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.301+0000 7fbe94b85640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbe7406d990 0x7fbe7406fe50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009b40 tx=0x7fbe84009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:30.303 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.301+0000 7fbe8d7fa640 10 client.4276.objecter ms_handle_connect 0x7fbe7406d990 2024-03-25T02:00:30.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.302+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4857+0+0 (secure 0 0 0) 0x7fbe88082c20 con 0x7fbe90146d90 2024-03-25T02:00:30.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.302+0000 7fbe8d7fa640 10 client.4276.objecter ms_dispatch 0x7fbe90000b90 osd_map(21..21 src has 1..21) 2024-03-25T02:00:30.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.302+0000 7fbe8d7fa640 3 client.4276.objecter handle_osd_map got epochs [21,21] > 0 2024-03-25T02:00:30.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.302+0000 7fbe8d7fa640 3 client.4276.objecter handle_osd_map decoding full epoch 21 2024-03-25T02:00:30.304 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.302+0000 7fbe8d7fa640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T02:00:30.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.312+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbe8807c690 con 0x7fbe90146d90 2024-03-25T02:00:30.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.312+0000 7fbe8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:30.314 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.312+0000 7fbe8d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:30.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.499+0000 7fbe96de9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:30.501 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.499+0000 7fbe96de9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:30.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.499+0000 7fbe96de9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.502 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.499+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe901d82c0 con 0x7fbe90146d90 2024-03-25T02:00:30.792 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.791+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fbe88005d10 con 0x7fbe90146d90 2024-03-25T02:00:30.795 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.794+0000 7fbe8d7fa640 1 -- 172.21.15.83:0/1980073417 <== mon.1 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+12885 (secure 0 0 0) 0x7fbe88089270 con 0x7fbe90146d90 2024-03-25T02:00:30.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.794+0000 7fbe8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:30.796 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.794+0000 7fbe8d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:30.796 INFO:teuthology.misc.health.smithi083.stdout: 2024-03-25T02:00:30.796 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":21,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T02:00:25.440849+0000","last_up_change":"2024-03-25T02:00:23.723108+0000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-25T02:00:14.575846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6801","nonce":2678950219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6803","nonce":2678950219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6807","nonce":2678950219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6805","nonce":2678950219}]},"public_addr":"172.21.15.83:6801/2678950219","cluster_addr":"172.21.15.83:6803/2678950219","heartbeat_back_addr":"172.21.15.83:6807/2678950219","heartbeat_front_addr":"172.21.15.83:6805/2678950219","state":["exists","up"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6825","nonce":2597989034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6827","nonce":2597989034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6831","nonce":2597989034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6829","nonce":2597989034}]},"public_addr":"172.21.15.83:6825/2597989034","cluster_addr":"172.21.15.83:6827/2597989034","heartbeat_back_addr":"172.21.15.83:6831/2597989034","heartbeat_front_addr":"172.21.15.83:6829/2597989034","state":["exists","up"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6817","nonce":602002618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6819","nonce":602002618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6823","nonce":602002618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6821","nonce":602002618}]},"public_addr":"172.21.15.83:6817/602002618","cluster_addr":"172.21.15.83:6819/602002618","heartbeat_back_addr":"172.21.15.83:6823/602002618","heartbeat_front_addr":"172.21.15.83:6821/602002618","state":["exists","up"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6809","nonce":3432501411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6811","nonce":3432501411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6815","nonce":3432501411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6813","nonce":3432501411}]},"public_addr":"172.21.15.83:6809/3432501411","cluster_addr":"172.21.15.83:6811/3432501411","heartbeat_back_addr":"172.21.15.83:6815/3432501411","heartbeat_front_addr":"172.21.15.83:6813/3432501411","state":["exists","up"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6819","nonce":2579144563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6821","nonce":2579144563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6825","nonce":2579144563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6823","nonce":2579144563}]},"public_addr":"172.21.15.153:6819/2579144563","cluster_addr":"172.21.15.153:6821/2579144563","heartbeat_back_addr":"172.21.15.153:6825/2579144563","heartbeat_front_addr":"172.21.15.153:6823/2579144563","state":["exists","up"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6811","nonce":2458137144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6813","nonce":2458137144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6817","nonce":2458137144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6815","nonce":2458137144}]},"public_addr":"172.21.15.153:6811/2458137144","cluster_addr":"172.21.15.153:6813/2458137144","heartbeat_back_addr":"172.21.15.153:6817/2458137144","heartbeat_front_addr":"172.21.15.153:6815/2458137144","state":["exists","up"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6803","nonce":4205760170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6805","nonce":4205760170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6809","nonce":4205760170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6807","nonce":4205760170}]},"public_addr":"172.21.15.153:6803/4205760170","cluster_addr":"172.21.15.153:6805/4205760170","heartbeat_back_addr":"172.21.15.153:6809/4205760170","heartbeat_front_addr":"172.21.15.153:6807/4205760170","state":["exists","up"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6827","nonce":2580901001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6829","nonce":2580901001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6833","nonce":2580901001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6831","nonce":2580901001}]},"public_addr":"172.21.15.153:6827/2580901001","cluster_addr":"172.21.15.153:6829/2580901001","heartbeat_back_addr":"172.21.15.153:6833/2580901001","heartbeat_front_addr":"172.21.15.153:6831/2580901001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.217280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.797510+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.322329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.630439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.336792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:09.768567+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.625955+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.445663+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:00:30.798 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 10 librados: watch_flush enter 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 10 librados: watch_flush exit 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 10 client.4276.objecter successfully canceled tick 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbe7406d990 msgr2=0x7fbe7406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbe7406d990 0x7fbe7406fe50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009b40 tx=0x7fbe84009190 comp rx=0 tx=0).stop 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 10 monclient: shutdown 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe90146d90 msgr2=0x7fbe901d8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe90146d90 0x7fbe901d8610 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe88009eb0 tx=0x7fbe88004a40 comp rx=0 tx=0).stop 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 shutdown_connections 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbe7406d990 0x7fbe7406fe50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe90146d90 0x7fbe901d8610 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe90140890 0x7fbe900b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.799 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 --2- 172.21.15.83:0/1980073417 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbe900b38e0 0x7fbe900b7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 >> 172.21.15.83:0/1980073417 conn(0x7fbe90067460 msgr2=0x7fbe9013dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:30.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 shutdown_connections 2024-03-25T02:00:30.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 -- 172.21.15.83:0/1980073417 wait complete. 2024-03-25T02:00:30.800 INFO:teuthology.misc.health.smithi083.stderr:2024-03-25T02:00:30.796+0000 7fbe96de9640 1 librados: shutdown 2024-03-25T02:00:30.808 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-25T02:00:30.808 INFO:tasks.ceph:Creating RBD pool 2024-03-25T02:00:30.809 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.937+0000 7fa690281640 10 monclient: get_monmap_and_config 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.937+0000 7fa690281640 10 monclient: build_initial_monmap 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.937+0000 7fa690281640 10 monclient: monmap: 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:30.938591+0000 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:30.938591+0000 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:30.939 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:30.940 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:30.940 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:30.940 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:30.940 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.937+0000 7fa690281640 1 Processor -- start 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.937+0000 7fa690281640 1 -- start start 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.937+0000 7fa690281640 10 monclient: init 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa6880a9b00 0x7fa688143700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient(hunting): picked mon.noname-a con 0x7fa6880a9b00 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa6880a9ee0 0x7fa6881401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient(hunting): picked mon.noname-c con 0x7fa6880a9ee0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa688143c40 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6880aba70 con 0x7fa688143c40 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa6880ab820 con 0x7fa6880a9b00 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fa6880abd50 con 0x7fa6880a9ee0 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:30.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.938+0000 7fa690281640 10 monclient(hunting): authenticate will time out at 1350.128497s 2024-03-25T02:00:30.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.944+0000 7fa68e81e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa688143c40 0x7fa688140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:49528/0 (socket says 172.21.15.83:49528) 2024-03-25T02:00:30.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.944+0000 7fa68e81e640 1 -- 172.21.15.83:0/4040943747 learned_addr learned my addr 172.21.15.83:0/4040943747 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:30.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.944+0000 7fa68d81c640 1 --2- 172.21.15.83:0/4040943747 >> v2:172.21.15.153:3300/0 conn(0x7fa6880a9ee0 0x7fa6881401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.945+0000 7fa68d81c640 10 monclient(hunting): get_auth_request con 0x7fa6880a9ee0 auth_method 0 2024-03-25T02:00:30.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.945+0000 7fa68d81c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.945+0000 7fa68d81c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.945+0000 7fa68d81c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.946+0000 7fa68d81c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.946+0000 7fa68d81c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.946+0000 7fa68d81c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 1 -- 172.21.15.83:0/4040943747 >> v2:172.21.15.83:3300/0 conn(0x7fa6880a9b00 msgr2=0x7fa688143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 1 --2- 172.21.15.83:0/4040943747 >> v2:172.21.15.83:3300/0 conn(0x7fa6880a9b00 0x7fa688143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 1 -- 172.21.15.83:0/4040943747 >> v1:172.21.15.153:6789/0 conn(0x7fa688143c40 legacy=0x7fa688140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 10 monclient: _finish_hunting 0 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 1 monclient: found mon.noname-c 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.949+0000 7fa68d81c640 1 -- 172.21.15.83:0/4040943747 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6880ac030 con 0x7fa6880a9ee0 2024-03-25T02:00:30.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.952+0000 7fa68d81c640 1 --2- 172.21.15.83:0/4040943747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa6881401f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6880b99f0 tx=0x7fa67802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0780f4e728c5a65 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:30.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 1 -- 172.21.15.83:0/4040943747 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678035020 con 0x7fa6880a9ee0 2024-03-25T02:00:30.957 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: dump: 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:30.958 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: _finish_auth 0 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: _check_auth_tickets 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 1 -- 172.21.15.83:0/4040943747 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa678031830 con 0x7fa6880a9ee0 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa690281640 5 monclient: authenticate success, global_id 4279 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa690281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 1 -- 172.21.15.83:0/4040943747 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678031c00 con 0x7fa6880a9ee0 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa690281640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa690281640 4 set_mon_vals no callback set 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.955+0000 7fa68d01b640 10 monclient: dump: 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.959 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.960 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.960 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.960 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.960 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.960 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.960 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.960 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 10 monclient: shutdown 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 -- 172.21.15.83:0/4040943747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 msgr2=0x7fa6881401f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 --2- 172.21.15.83:0/4040943747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa6881401f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6880b99f0 tx=0x7fa67802b8d0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 -- 172.21.15.83:0/4040943747 shutdown_connections 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 --2- 172.21.15.83:0/4040943747 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa6881401f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 --2- 172.21.15.83:0/4040943747 >> v2:172.21.15.83:3300/0 conn(0x7fa6880a9b00 0x7fa688143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 -- 172.21.15.83:0/4040943747 >> 172.21.15.83:0/4040943747 conn(0x7fa6880b8990 msgr2=0x7fa6880b8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 -- 172.21.15.83:0/4040943747 shutdown_connections 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.956+0000 7fa690281640 1 -- 172.21.15.83:0/4040943747 wait complete. 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 10 monclient: build_initial_monmap 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 10 monclient: monmap: 2024-03-25T02:00:30.964 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 1 librados: starting msgr at 2024-03-25T02:00:30.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 1 librados: starting objecter 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 1 Processor -- start 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 1 -- start start 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 1 librados: setting wanted keys 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 1 librados: calling monclient init 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.957+0000 7fa690281640 10 monclient: init 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa6880a9b00 0x7fa688136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): picked mon.noname-c con 0x7fa6880a9b00 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa68813b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): picked mon.noname-b con 0x7fa6880a9ee0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa688143c40 0x7fa688139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): picked mon.noname-a con 0x7fa688143c40 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:30.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa6880b3490 con 0x7fa688143c40 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6880b3310 con 0x7fa6880a9ee0 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fa68814d9c0 con 0x7fa6880a9b00 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa690281640 10 monclient(hunting): authenticate will time out at 1350.148436s 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.958+0000 7fa68d81c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa68813b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa6880a9b00 0x7fa688136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa68813b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:54336/0 (socket says 172.21.15.83:54336) 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 1 -- 172.21.15.83:0/3462347707 learned_addr learned my addr 172.21.15.83:0/3462347707 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 10 monclient(hunting): get_auth_request con 0x7fa6880a9ee0 auth_method 0 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 10 monclient(hunting): get_auth_request con 0x7fa6880a9b00 auth_method 0 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68d81c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.959+0000 7fa68e01d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 1 -- 172.21.15.83:0/3462347707 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa6880a9b00 msgr2=0x7fa688136f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa6880a9b00 0x7fa688136f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 1 -- 172.21.15.83:0/3462347707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa688143c40 msgr2=0x7fa688139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa688143c40 0x7fa688139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 10 monclient: _finish_hunting 0 2024-03-25T02:00:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 1 monclient: found mon.noname-b 2024-03-25T02:00:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.969 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.962+0000 7fa68d81c640 1 -- 172.21.15.83:0/3462347707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68813d100 con 0x7fa6880a9ee0 2024-03-25T02:00:30.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.964+0000 7fa68d81c640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa68813b480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa678009890 tx=0x7fa6780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.964+0000 7fa67effd640 10 client.?.objecter ms_handle_connect 0x7fa6880a9ee0 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.964+0000 7fa67effd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 1 -- 172.21.15.83:0/3462347707 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678041070 con 0x7fa6880a9ee0 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: dump: 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.971 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: _finish_auth 0 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 1 -- 172.21.15.83:0/3462347707 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa67803a950 con 0x7fa6880a9ee0 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 1 -- 172.21.15.83:0/3462347707 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa678035040 con 0x7fa6880a9ee0 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa67effd640 10 monclient: dump: 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.972 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 5 monclient: authenticate success, global_id 4282 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 10 monclient: _renew_subs 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa68c81a640 4 set_mon_vals no callback set 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa688138ca0 con 0x7fa6880a9ee0 2024-03-25T02:00:30.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 10 monclient: _renew_subs 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6881399c0 con 0x7fa6880a9ee0 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.967+0000 7fa690281640 1 librados: init done 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.968+0000 7fa690281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.968+0000 7fa690281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.968+0000 7fa690281640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:30.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.968+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa688138aa0 con 0x7fa6880a9ee0 2024-03-25T02:00:30.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.986+0000 7fa67effd640 1 -- 172.21.15.83:0/3462347707 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fa678038020 con 0x7fa6880a9ee0 2024-03-25T02:00:30.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.986+0000 7fa67effd640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa66c06da70 0x7fa66c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:30.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.987+0000 7fa68e01d640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa66c06da70 0x7fa66c06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:30.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.987+0000 7fa68e01d640 10 monclient: get_auth_request con 0x7fa66c06da70 auth_method 0 2024-03-25T02:00:30.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.987+0000 7fa68e01d640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa66c06da70 0x7fa66c06ff30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa684009990 tx=0x7fa684009110 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:30.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:30.987+0000 7fa67effd640 10 client.4282.objecter ms_handle_connect 0x7fa66c06da70 2024-03-25T02:00:31.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.214+0000 7fa67effd640 1 -- 172.21.15.83:0/3462347707 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 4857+0+0 (secure 0 0 0) 0x7fa678081af0 con 0x7fa6880a9ee0 2024-03-25T02:00:31.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.214+0000 7fa67effd640 10 client.4282.objecter ms_dispatch 0x7fa688000b90 osd_map(21..21 src has 1..21) 2024-03-25T02:00:31.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.214+0000 7fa67effd640 3 client.4282.objecter handle_osd_map got epochs [21,21] > 0 2024-03-25T02:00:31.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.214+0000 7fa67effd640 3 client.4282.objecter handle_osd_map decoding full epoch 21 2024-03-25T02:00:31.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.214+0000 7fa67effd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T02:00:31.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.225+0000 7fa67effd640 1 -- 172.21.15.83:0/3462347707 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa67807bbb0 con 0x7fa6880a9ee0 2024-03-25T02:00:31.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.225+0000 7fa67effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:31.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.225+0000 7fa67effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:31.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.407+0000 7fa690281640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-25T02:00:31.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.407+0000 7fa690281640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-25T02:00:31.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.407+0000 7fa690281640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:31.409 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.407+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fa68804f940 con 0x7fa6880a9ee0 2024-03-25T02:00:31.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.997+0000 7fa67effd640 1 -- 172.21.15.83:0/3462347707 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v22) ==== 109+0+0 (secure 0 0 0) 0x7fa67807b4b0 con 0x7fa6880a9ee0 2024-03-25T02:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.997+0000 7fa67effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-03-25T02:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:31.997+0000 7fa67effd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-03-25T02:00:32.000 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'rbd' created 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 10 librados: watch_flush enter 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 10 librados: watch_flush exit 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 10 client.4282.objecter successfully canceled tick 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa66c06da70 msgr2=0x7fa66c06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa66c06da70 0x7fa66c06ff30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa684009990 tx=0x7fa684009110 comp rx=0 tx=0).stop 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 10 monclient: shutdown 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 msgr2=0x7fa68813b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa68813b480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa678009890 tx=0x7fa6780046c0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 shutdown_connections 2024-03-25T02:00:32.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa66c06da70 0x7fa66c06ff30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa688143c40 0x7fa688139480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6880a9ee0 0x7fa68813b480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 --2- 172.21.15.83:0/3462347707 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa6880a9b00 0x7fa688136f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 >> 172.21.15.83:0/3462347707 conn(0x7fa6880b8990 msgr2=0x7fa6880b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:32.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 shutdown_connections 2024-03-25T02:00:32.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 -- 172.21.15.83:0/3462347707 wait complete. 2024-03-25T02:00:32.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.000+0000 7fa690281640 1 librados: shutdown 2024-03-25T02:00:32.017 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.118+0000 7f4d51d24640 10 monclient: get_monmap_and_config 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.118+0000 7f4d51d24640 10 monclient: build_initial_monmap 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.118+0000 7f4d51d24640 10 monclient: monmap: 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:32.119616+0000 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:32.119616+0000 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:32.120 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:32.121 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:32.121 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:32.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.118+0000 7f4d51d24640 1 Processor -- start 2024-03-25T02:00:32.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.118+0000 7f4d51d24640 1 -- start start 2024-03-25T02:00:32.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.118+0000 7f4d51d24640 10 monclient: init 2024-03-25T02:00:32.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:32.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-25T02:00:32.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f4d4c0af4a0 0x7f4d4c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:32.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4c0af4a0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d4c13d560 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4d4c06fb00 0x7f4d4c14a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient(hunting): picked mon.noname-a con 0x7f4d4c06fb00 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d4c0725b0 con 0x7f4d4c13d560 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f4d4c0af880 con 0x7f4d4c06fb00 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f4d4c072820 con 0x7f4d4c0af4a0 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.119+0000 7f4d51d24640 10 monclient(hunting): authenticate will time out at 1351.309497s 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4d4c06fb00 0x7f4d4c14a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f4d4c06fb00 0x7f4d4c14a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41078/0 (socket says 172.21.15.83:41078) 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 1 -- 172.21.15.83:0/3802004018 learned_addr learned my addr 172.21.15.83:0/3802004018 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:32.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 10 monclient(hunting): get_auth_request con 0x7f4d4c06fb00 auth_method 0 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4b7fe640 1 --2- 172.21.15.83:0/3802004018 >> v2:172.21.15.153:3300/0 conn(0x7f4d4c0af4a0 0x7f4d4c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.120+0000 7f4d4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.121+0000 7f4d4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.121+0000 7f4d4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.121+0000 7f4d4b7fe640 10 monclient(hunting): get_auth_request con 0x7f4d4c0af4a0 auth_method 0 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.121+0000 7f4d4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.121+0000 7f4d4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:32.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.121+0000 7f4d4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:32.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 1 -- 172.21.15.83:0/3802004018 >> v2:172.21.15.153:3300/0 conn(0x7f4d4c0af4a0 msgr2=0x7f4d4c13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 1 --2- 172.21.15.83:0/3802004018 >> v2:172.21.15.153:3300/0 conn(0x7f4d4c0af4a0 0x7f4d4c13d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 1 -- 172.21.15.83:0/3802004018 >> v1:172.21.15.153:6789/0 conn(0x7f4d4c13d560 legacy=0x7f4d4c06f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 1 monclient: found mon.noname-a 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.122+0000 7f4d4bfff640 1 -- 172.21.15.83:0/3802004018 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c072b00 con 0x7f4d4c06fb00 2024-03-25T02:00:32.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.124+0000 7f4d4bfff640 1 --2- 172.21.15.83:0/3802004018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c14a690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c06d7e0 tx=0x7f4d4402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=115540062267e72a server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 1 -- 172.21.15.83:0/3802004018 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4402c9f0 con 0x7f4d4c06fb00 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: dump: 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:32.129 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:32.130 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:32.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:32.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:32.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:32.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d51d24640 5 monclient: authenticate success, global_id 4269 2024-03-25T02:00:32.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d51d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:32.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 1 -- 172.21.15.83:0/3802004018 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d4402cb90 con 0x7f4d4c06fb00 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 1 -- 172.21.15.83:0/3802004018 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d440343f0 con 0x7f4d4c06fb00 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d4a7fc640 10 monclient: dump: 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:32.132 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d51d24640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.127+0000 7f4d51d24640 4 set_mon_vals no callback set 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 10 monclient: shutdown 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 -- 172.21.15.83:0/3802004018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c06fb00 msgr2=0x7f4d4c14a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3802004018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c14a690 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c06d7e0 tx=0x7f4d4402be50 comp rx=0 tx=0).stop 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 -- 172.21.15.83:0/3802004018 shutdown_connections 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3802004018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c14a690 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3802004018 >> v2:172.21.15.153:3300/0 conn(0x7f4d4c0af4a0 0x7f4d4c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 -- 172.21.15.83:0/3802004018 >> 172.21.15.83:0/3802004018 conn(0x7f4d4c0b4a10 msgr2=0x7f4d4c0b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 -- 172.21.15.83:0/3802004018 shutdown_connections 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.128+0000 7f4d51d24640 1 -- 172.21.15.83:0/3802004018 wait complete. 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.129+0000 7f4d51d24640 10 monclient: build_initial_monmap 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.129+0000 7f4d51d24640 10 monclient: monmap: 2024-03-25T02:00:32.133 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.130+0000 7f4d51d24640 1 librados: starting msgr at 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.130+0000 7f4d51d24640 1 librados: starting objecter 2024-03-25T02:00:32.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.130+0000 7f4d51d24640 1 Processor -- start 2024-03-25T02:00:32.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.130+0000 7f4d51d24640 1 -- start start 2024-03-25T02:00:32.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.130+0000 7f4d51d24640 1 librados: setting wanted keys 2024-03-25T02:00:32.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.130+0000 7f4d51d24640 1 librados: calling monclient init 2024-03-25T02:00:32.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.130+0000 7f4d51d24640 10 monclient: init 2024-03-25T02:00:32.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:32.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:00:32.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c13fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): picked mon.noname-b con 0x7f4d4c06fb00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c0af4a0 0x7f4d4c133dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): picked mon.noname-a con 0x7f4d4c0af4a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f4d4c13d560 0x7f4d4c132c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4c13d560 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c13d980 con 0x7f4d4c0af4a0 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c070a20 con 0x7f4d4c06fb00 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4c070f40 con 0x7f4d4c13d560 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:32.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.131+0000 7f4d51d24640 10 monclient(hunting): authenticate will time out at 1351.321483s 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c13fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4bfff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f4d4c13d560 0x7f4d4c132c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c0af4a0 0x7f4d4c133dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c13fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:54348/0 (socket says 172.21.15.83:54348) 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 1 -- 172.21.15.83:0/3165539036 learned_addr learned my addr 172.21.15.83:0/3165539036 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 10 monclient(hunting): get_auth_request con 0x7f4d4c06fb00 auth_method 0 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4affd640 10 monclient(hunting): get_auth_request con 0x7f4d4c0af4a0 auth_method 0 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:32.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4bfff640 10 monclient(hunting): get_auth_request con 0x7f4d4c13d560 auth_method 0 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.132+0000 7f4d4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.133+0000 7f4d4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.133+0000 7f4d4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.133+0000 7f4d4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.133+0000 7f4d4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.133+0000 7f4d4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.133+0000 7f4d4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 10 monclient(hunting): handle_auth_done global_id 4291 payload 824 2024-03-25T02:00:32.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 1 -- 172.21.15.83:0/3165539036 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f4d4c13d560 msgr2=0x7f4d4c132c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f4d4c13d560 0x7f4d4c132c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 1 -- 172.21.15.83:0/3165539036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c0af4a0 msgr2=0x7f4d4c133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c0af4a0 0x7f4d4c133dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 1 monclient: found mon.noname-b 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:32.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.135+0000 7f4d4b7fe640 1 -- 172.21.15.83:0/3165539036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c133150 con 0x7f4d4c06fb00 2024-03-25T02:00:32.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.137+0000 7f4d4b7fe640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c13fec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000e9e0 tx=0x7f4d4000eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:32.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.137+0000 7f4d33fff640 10 client.?.objecter ms_handle_connect 0x7f4d4c06fb00 2024-03-25T02:00:32.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.137+0000 7f4d33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 1 -- 172.21.15.83:0/3165539036 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40017070 con 0x7f4d4c06fb00 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: dump: 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:32.144 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: _finish_auth 0 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 1 -- 172.21.15.83:0/3165539036 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d4000c9d0 con 0x7f4d4c06fb00 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 1 -- 172.21.15.83:0/3165539036 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000cdc0 con 0x7f4d4c06fb00 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 5 monclient: authenticate success, global_id 4291 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 10 monclient: _renew_subs 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4c133410 con 0x7f4d4c06fb00 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:32.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d33fff640 10 monclient: dump: 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d497fa640 4 set_mon_vals no callback set 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 10 monclient: _renew_subs 2024-03-25T02:00:32.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4c135800 con 0x7f4d4c06fb00 2024-03-25T02:00:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.141+0000 7f4d51d24640 1 librados: init done 2024-03-25T02:00:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.142+0000 7f4d51d24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.142+0000 7f4d51d24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.142+0000 7f4d51d24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:32.147 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.142+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d4c068eb0 con 0x7f4d4c06fb00 2024-03-25T02:00:32.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.170+0000 7f4d33fff640 1 -- 172.21.15.83:0/3165539036 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f4d4001a070 con 0x7f4d4c06fb00 2024-03-25T02:00:32.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.171+0000 7f4d33fff640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f4d3806da20 0x7f4d3806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:32.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.171+0000 7f4d4affd640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f4d3806da20 0x7f4d3806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:32.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.171+0000 7f4d4affd640 10 monclient: get_auth_request con 0x7f4d3806da20 auth_method 0 2024-03-25T02:00:32.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.171+0000 7f4d33fff640 10 client.4291.objecter ms_handle_connect 0x7f4d3806da20 2024-03-25T02:00:32.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.171+0000 7f4d4affd640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f4d3806da20 0x7f4d3806fee0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c138f10 tx=0x7f4d3c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:32.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.176+0000 7f4d33fff640 1 -- 172.21.15.83:0/3165539036 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5199+0+0 (secure 0 0 0) 0x7f4d40057b20 con 0x7f4d4c06fb00 2024-03-25T02:00:32.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.176+0000 7f4d33fff640 10 client.4291.objecter ms_dispatch 0x7f4d4c000b90 osd_map(22..22 src has 1..22) 2024-03-25T02:00:32.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.176+0000 7f4d33fff640 3 client.4291.objecter handle_osd_map got epochs [22,22] > 0 2024-03-25T02:00:32.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.176+0000 7f4d33fff640 3 client.4291.objecter handle_osd_map decoding full epoch 22 2024-03-25T02:00:32.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.176+0000 7f4d33fff640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T02:00:32.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.439+0000 7f4d33fff640 1 -- 172.21.15.83:0/3165539036 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4d4c068eb0 con 0x7f4d4c06fb00 2024-03-25T02:00:32.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.439+0000 7f4d33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:32.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.439+0000 7f4d33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:32.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.639+0000 7f4d51d24640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-25T02:00:32.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.639+0000 7f4d51d24640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-25T02:00:32.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.639+0000 7f4d51d24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:32.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:32.639+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f4d4c070640 con 0x7f4d4c06fb00 2024-03-25T02:00:33.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.138+0000 7f4d33fff640 1 -- 172.21.15.83:0/3165539036 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v23) ==== 173+0+0 (secure 0 0 0) 0x7f4d4c070640 con 0x7f4d4c06fb00 2024-03-25T02:00:33.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.138+0000 7f4d33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-03-25T02:00:33.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.138+0000 7f4d33fff640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-03-25T02:00:33.141 INFO:teuthology.orchestra.run.smithi083.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-25T02:00:33.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 10 librados: watch_flush enter 2024-03-25T02:00:33.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 10 librados: watch_flush exit 2024-03-25T02:00:33.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T02:00:33.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 10 client.4291.objecter successfully canceled tick 2024-03-25T02:00:33.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f4d3806da20 msgr2=0x7f4d3806fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f4d3806da20 0x7f4d3806fee0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c138f10 tx=0x7f4d3c009040 comp rx=0 tx=0).stop 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 10 monclient: shutdown 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c06fb00 msgr2=0x7f4d4c13fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c13fec0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000e9e0 tx=0x7f4d4000eea0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 shutdown_connections 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f4d3806da20 0x7f4d3806fee0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f4d4c13d560 0x7f4d4c132c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d4c0af4a0 0x7f4d4c133dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.140+0000 7f4d51d24640 1 --2- 172.21.15.83:0/3165539036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d4c06fb00 0x7f4d4c13fec0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.141+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 >> 172.21.15.83:0/3165539036 conn(0x7f4d4c0b4a10 msgr2=0x7f4d4c0ae7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.141+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 shutdown_connections 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.141+0000 7f4d51d24640 1 -- 172.21.15.83:0/3165539036 wait complete. 2024-03-25T02:00:33.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.141+0000 7f4d51d24640 1 librados: shutdown 2024-03-25T02:00:33.154 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-25T02:00:33.154 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-25T02:00:33.155 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-25T02:00:33.155 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-25T02:00:33.155 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-25T02:00:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.254+0000 7f37375c8640 10 monclient: get_monmap_and_config 2024-03-25T02:00:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.254+0000 7f37375c8640 10 monclient: build_initial_monmap 2024-03-25T02:00:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.255+0000 7f37375c8640 10 monclient: monmap: 2024-03-25T02:00:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:33.257 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:33.256397+0000 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:33.256397+0000 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.255+0000 7f37375c8640 1 Processor -- start 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.255+0000 7f37375c8640 1 -- start start 2024-03-25T02:00:33.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.255+0000 7f37375c8640 10 monclient: init 2024-03-25T02:00:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-03-25T02:00:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37300b04a0 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:33.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37300b2df0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3730147350 0x7f373014eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient(hunting): picked mon.noname-c con 0x7f3730147350 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37300b2410 con 0x7f37300b04a0 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37300b26f0 con 0x7f37300b2df0 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f37300b29d0 con 0x7f3730147350 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.256+0000 7f37375c8640 10 monclient(hunting): authenticate will time out at 1352.446359s 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.257+0000 7f3735364640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f37300b04a0 0x7f3730146c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:59398/0 (socket says 172.21.15.83:59398) 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.257+0000 7f3735364640 1 -- 172.21.15.83:0/427886058 learned_addr learned my addr 172.21.15.83:0/427886058 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.257+0000 7f3735b65640 1 --2- 172.21.15.83:0/427886058 >> v2:172.21.15.153:3300/0 conn(0x7f3730147350 0x7f373014eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.257+0000 7f3735b65640 10 monclient(hunting): get_auth_request con 0x7f3730147350 auth_method 0 2024-03-25T02:00:33.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.257+0000 7f3735b65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.257+0000 7f3735b65640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.257+0000 7f3735b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.258+0000 7f3735b65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.258+0000 7f3735b65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:33.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.258+0000 7f3735b65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.259+0000 7f3727fff640 1 -- 172.21.15.83:0/427886058 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362154252 0 0) 0x7f37300b2410 con 0x7f37300b04a0 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.259+0000 7f3727fff640 10 monclient(hunting): _negotiate 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.259+0000 7f3727fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.259+0000 7f3727fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.259+0000 7f3727fff640 10 monclient(hunting): my global_id is 4275 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.259+0000 7f3727fff640 1 -- 172.21.15.83:0/427886058 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f371c003520 con 0x7f37300b04a0 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 1 -- 172.21.15.83:0/427886058 >> v1:172.21.15.153:6789/0 conn(0x7f37300b2df0 legacy=0x7f3730140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:33.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 1 -- 172.21.15.83:0/427886058 >> v1:172.21.15.83:6789/0 conn(0x7f37300b04a0 legacy=0x7f3730146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 10 monclient: _finish_hunting 0 2024-03-25T02:00:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 1 monclient: found mon.noname-c 2024-03-25T02:00:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:33.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.260+0000 7f3735b65640 1 -- 172.21.15.83:0/427886058 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37300b3380 con 0x7f3730147350 2024-03-25T02:00:33.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.266+0000 7f3735b65640 1 --2- 172.21.15.83:0/427886058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3730147350 0x7f373014eb00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3728009dc0 tx=0x7f372802bf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89eb5a854d5c9275 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:33.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 1 -- 172.21.15.83:0/427886058 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728039070 con 0x7f3730147350 2024-03-25T02:00:33.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:33.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:33.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: dump: 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: _finish_auth 0 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:33.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 1 -- 172.21.15.83:0/427886058 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f372802ccd0 con 0x7f3730147350 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 1 -- 172.21.15.83:0/427886058 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728034660 con 0x7f3730147350 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f3727fff640 10 monclient: dump: 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:33.274 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:33.275 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:33.275 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:33.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f37375c8640 5 monclient: authenticate success, global_id 4294 2024-03-25T02:00:33.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f37375c8640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:33.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.270+0000 7f37375c8640 4 set_mon_vals no callback set 2024-03-25T02:00:33.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 10 monclient: shutdown 2024-03-25T02:00:33.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:33.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 1 -- 172.21.15.83:0/427886058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3730147350 msgr2=0x7f373014eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:33.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 1 --2- 172.21.15.83:0/427886058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3730147350 0x7f373014eb00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3728009dc0 tx=0x7f372802bf20 comp rx=0 tx=0).stop 2024-03-25T02:00:33.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 1 -- 172.21.15.83:0/427886058 shutdown_connections 2024-03-25T02:00:33.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 1 --2- 172.21.15.83:0/427886058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3730147350 0x7f373014eb00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 1 -- 172.21.15.83:0/427886058 >> 172.21.15.83:0/427886058 conn(0x7f37300b8a00 msgr2=0x7f37300b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 1 -- 172.21.15.83:0/427886058 shutdown_connections 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.271+0000 7f37375c8640 1 -- 172.21.15.83:0/427886058 wait complete. 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 10 monclient: build_initial_monmap 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 10 monclient: monmap: 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:33.279 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 librados: starting msgr at 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 librados: starting objecter 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 Processor -- start 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 -- start start 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 librados: setting wanted keys 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 librados: calling monclient init 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 10 monclient: init 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37300b04a0 0x7f37301439c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 10 monclient(hunting): picked mon.noname-b con 0x7f37300b04a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:33.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37300b2df0 0x7f373013c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.272+0000 7f37375c8640 10 monclient(hunting): picked mon.noname-c con 0x7f37300b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3730147350 0x7f373013a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 10 monclient(hunting): picked mon.noname-a con 0x7f3730147350 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3730140c90 con 0x7f3730147350 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3730140b10 con 0x7f37300b04a0 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f3730140e10 con 0x7f37300b2df0 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f37375c8640 10 monclient(hunting): authenticate will time out at 1352.462993s 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735b65640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3730147350 0x7f373013a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735364640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37300b04a0 0x7f37301439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3734b63640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37300b2df0 0x7f373013c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:33.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735364640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37300b04a0 0x7f37301439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:54370/0 (socket says 172.21.15.83:54370) 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735364640 1 -- 172.21.15.83:0/1970111762 learned_addr learned my addr 172.21.15.83:0/1970111762 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735364640 10 monclient(hunting): get_auth_request con 0x7f37300b04a0 auth_method 0 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735364640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735364640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735364640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735b65640 10 monclient(hunting): get_auth_request con 0x7f3730147350 auth_method 0 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735b65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735b65640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3735b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.273+0000 7f3734b63640 10 monclient(hunting): get_auth_request con 0x7f37300b2df0 auth_method 0 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3734b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3734b63640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3734b63640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3735364640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:33.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3735364640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:33.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3735364640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:33.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3735b65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:33.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3735b65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:33.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3735b65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:33.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3734b63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:33.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3734b63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:33.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.274+0000 7f3734b63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 1 -- 172.21.15.83:0/1970111762 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37300b2df0 msgr2=0x7f373013c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37300b2df0 0x7f373013c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 1 -- 172.21.15.83:0/1970111762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3730147350 msgr2=0x7f373013a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3730147350 0x7f373013a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 10 monclient: _finish_hunting 0 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 1 monclient: found mon.noname-b 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:33.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.277+0000 7f3735364640 1 -- 172.21.15.83:0/1970111762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373013c910 con 0x7f37300b04a0 2024-03-25T02:00:33.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.280+0000 7f3735364640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37300b04a0 0x7f37301439c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f372000c460 tx=0x7f372000c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:33.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.280+0000 7f3725ffb640 10 client.?.objecter ms_handle_connect 0x7f37300b04a0 2024-03-25T02:00:33.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.280+0000 7f3725ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:33.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 1 -- 172.21.15.83:0/1970111762 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720017070 con 0x7f37300b04a0 2024-03-25T02:00:33.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: dump: 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:33.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 1 -- 172.21.15.83:0/1970111762 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3720005ce0 con 0x7f37300b04a0 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 1 -- 172.21.15.83:0/1970111762 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3720007290 con 0x7f37300b04a0 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 5 monclient: authenticate success, global_id 4300 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 10 monclient: _renew_subs 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373013ae70 con 0x7f37300b04a0 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37277fe640 4 set_mon_vals no callback set 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f3725ffb640 10 monclient: dump: 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:33.289 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 10 monclient: _renew_subs 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37301db8f0 con 0x7f37300b04a0 2024-03-25T02:00:33.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.285+0000 7f37375c8640 1 librados: init done 2024-03-25T02:00:33.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.286+0000 7f37375c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:33.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.286+0000 7f37375c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:33.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.286+0000 7f37375c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:33.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.286+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373013a240 con 0x7f37300b04a0 2024-03-25T02:00:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.316+0000 7f3725ffb640 1 -- 172.21.15.83:0/1970111762 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f372001a070 con 0x7f37300b04a0 2024-03-25T02:00:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.316+0000 7f3725ffb640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f371406da50 0x7f371406ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.316+0000 7f3734b63640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f371406da50 0x7f371406ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.316+0000 7f3734b63640 10 monclient: get_auth_request con 0x7f371406da50 auth_method 0 2024-03-25T02:00:33.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.317+0000 7f3725ffb640 10 client.4300.objecter ms_handle_connect 0x7f371406da50 2024-03-25T02:00:33.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.317+0000 7f3734b63640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f371406da50 0x7f371406ff10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3730138bd0 tx=0x7f372c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.321+0000 7f3725ffb640 1 -- 172.21.15.83:0/1970111762 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5210+0+0 (secure 0 0 0) 0x7f3720056a90 con 0x7f37300b04a0 2024-03-25T02:00:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.321+0000 7f3725ffb640 10 client.4300.objecter ms_dispatch 0x7f3730000b90 osd_map(23..23 src has 1..23) 2024-03-25T02:00:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.321+0000 7f3725ffb640 3 client.4300.objecter handle_osd_map got epochs [23,23] > 0 2024-03-25T02:00:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.321+0000 7f3725ffb640 3 client.4300.objecter handle_osd_map decoding full epoch 23 2024-03-25T02:00:33.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.321+0000 7f3725ffb640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T02:00:33.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.625+0000 7f3725ffb640 1 -- 172.21.15.83:0/1970111762 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3720021020 con 0x7f37300b04a0 2024-03-25T02:00:33.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.625+0000 7f3725ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:33.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.625+0000 7f3725ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:33.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.933+0000 7f37375c8640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-03-25T02:00:33.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.933+0000 7f37375c8640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-25T02:00:33.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.933+0000 7f37375c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:33.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:33.933+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3730075810 con 0x7f37300b04a0 2024-03-25T02:00:34.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.411+0000 7f3725ffb640 1 -- 172.21.15.83:0/1970111762 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+173180 (secure 0 0 0) 0x7f373013c910 con 0x7f37300b04a0 2024-03-25T02:00:34.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.411+0000 7f3725ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-03-25T02:00:34.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.411+0000 7f3725ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:34.416 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:00:34.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 10 librados: watch_flush enter 2024-03-25T02:00:34.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 10 librados: watch_flush exit 2024-03-25T02:00:34.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T02:00:34.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 10 client.4300.objecter successfully canceled tick 2024-03-25T02:00:34.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f371406da50 msgr2=0x7f371406ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:34.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f371406da50 0x7f371406ff10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3730138bd0 tx=0x7f372c009040 comp rx=0 tx=0).stop 2024-03-25T02:00:34.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 10 monclient: shutdown 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37300b04a0 msgr2=0x7f37301439c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37300b04a0 0x7f37301439c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f372000c460 tx=0x7f372000c920 comp rx=0 tx=0).stop 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 shutdown_connections 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f371406da50 0x7f371406ff10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3730147350 0x7f373013a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37300b2df0 0x7f373013c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 --2- 172.21.15.83:0/1970111762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37300b04a0 0x7f37301439c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 >> 172.21.15.83:0/1970111762 conn(0x7f37300b8a00 msgr2=0x7f37300b7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 shutdown_connections 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 -- 172.21.15.83:0/1970111762 wait complete. 2024-03-25T02:00:34.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.414+0000 7f37375c8640 1 librados: shutdown 2024-03-25T02:00:34.428 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":5,"flags":0,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4072599690},{"type":"v1","addr":"172.21.15.153:6801","nonce":4072599690}]},"active_addr":"172.21.15.153:6801/4072599690","active_change":"2024-03-25T01:59:34.007956+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2201632976}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3610363178}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2729589416}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1271499710}]}]} 2024-03-25T02:00:34.434 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-25T02:00:34.434 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-25T02:00:34.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.562+0000 7f6571a7b640 10 monclient: get_monmap_and_config 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.562+0000 7f6571a7b640 10 monclient: build_initial_monmap 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.562+0000 7f6571a7b640 10 monclient: monmap: 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:34.563513+0000 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:34.563513+0000 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:34.564 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:34.565 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:34.565 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:34.565 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:34.565 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:34.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.562+0000 7f6571a7b640 1 Processor -- start 2024-03-25T02:00:34.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.562+0000 7f6571a7b640 1 -- start start 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.562+0000 7f6571a7b640 10 monclient: init 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f656c0b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f656c0b2df0 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f656c140b10 0x7f656c147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f656c140b10 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656c0b2410 con 0x7f656c0b2df0 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f656c0b26f0 con 0x7f656c140b10 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:34.568 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f656c0b29d0 con 0x7f656c0b04a0 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.563+0000 7f6571a7b640 10 monclient(hunting): authenticate will time out at 1353.753489s 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.564+0000 7f656bfff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f656c140b10 0x7f656c147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.564+0000 7f656b7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.564+0000 7f656affd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f656c0b2df0 0x7f656c140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45584/0 (socket says 172.21.15.83:45584) 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656affd640 1 -- 172.21.15.83:0/3663008499 learned_addr learned my addr 172.21.15.83:0/3663008499 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656b7fe640 10 monclient(hunting): get_auth_request con 0x7f656c0b04a0 auth_method 0 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656bfff640 10 monclient(hunting): get_auth_request con 0x7f656c140b10 auth_method 0 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:34.569 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:34.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:34.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:34.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:34.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:34.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.565+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:34.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.566+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 1 -- 172.21.15.83:0/3663008499 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 msgr2=0x7f656c146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 1 --2- 172.21.15.83:0/3663008499 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c146c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 1 -- 172.21.15.83:0/3663008499 >> v1:172.21.15.83:6789/0 conn(0x7f656c0b2df0 legacy=0x7f656c140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 1 monclient: found mon.noname-a 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:34.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.567+0000 7f656bfff640 1 -- 172.21.15.83:0/3663008499 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c0b3380 con 0x7f656c140b10 2024-03-25T02:00:34.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.573+0000 7f656bfff640 1 --2- 172.21.15.83:0/3663008499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c140b10 0x7f656c147180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f655c009d90 tx=0x7f655c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e3aaa4422495c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:34.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 1 -- 172.21.15.83:0/3663008499 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c039070 con 0x7f656c140b10 2024-03-25T02:00:34.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:34.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:34.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: dump: 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:34.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 1 -- 172.21.15.83:0/3663008499 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655c02cce0 con 0x7f656c140b10 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f6571a7b640 5 monclient: authenticate success, global_id 4284 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f6571a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f6571a7b640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f6571a7b640 4 set_mon_vals no callback set 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 1 -- 172.21.15.83:0/3663008499 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c034660 con 0x7f656c140b10 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.577+0000 7f656a7fc640 10 monclient: dump: 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.581 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.582 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:34.582 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:34.582 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:34.582 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:34.582 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:34.582 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:34.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 10 monclient: shutdown 2024-03-25T02:00:34.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:34.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 -- 172.21.15.83:0/3663008499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c140b10 msgr2=0x7f656c147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:34.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 --2- 172.21.15.83:0/3663008499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c140b10 0x7f656c147180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f655c009d90 tx=0x7f655c02bf30 comp rx=0 tx=0).stop 2024-03-25T02:00:34.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 -- 172.21.15.83:0/3663008499 shutdown_connections 2024-03-25T02:00:34.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 --2- 172.21.15.83:0/3663008499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c140b10 0x7f656c147180 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.586 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 --2- 172.21.15.83:0/3663008499 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 -- 172.21.15.83:0/3663008499 >> 172.21.15.83:0/3663008499 conn(0x7f656c0b8a00 msgr2=0x7f656c0b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 -- 172.21.15.83:0/3663008499 shutdown_connections 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.578+0000 7f6571a7b640 1 -- 172.21.15.83:0/3663008499 wait complete. 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.579+0000 7f6571a7b640 10 monclient: build_initial_monmap 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.579+0000 7f6571a7b640 10 monclient: monmap: 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:34.587 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.580+0000 7f6571a7b640 1 librados: starting msgr at 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.580+0000 7f6571a7b640 1 librados: starting objecter 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.580+0000 7f6571a7b640 1 Processor -- start 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.580+0000 7f6571a7b640 1 -- start start 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.580+0000 7f6571a7b640 1 librados: setting wanted keys 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.580+0000 7f6571a7b640 1 librados: calling monclient init 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.580+0000 7f6571a7b640 10 monclient: init 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c1340b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:34.588 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f656c0b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c0b2df0 0x7f656c12dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f656c0b2df0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f656c140b10 0x7f656c12e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f656c140b10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f656c14dc70 con 0x7f656c0b2df0 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f656c14daf0 con 0x7f656c140b10 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f656c14ddf0 con 0x7f656c0b04a0 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:34.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f6571a7b640 10 monclient(hunting): authenticate will time out at 1353.771230s 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f656affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c0b2df0 0x7f656c12dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f656bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f656c140b10 0x7f656c12e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.581+0000 7f656b7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c1340b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f656c140b10 0x7f656c12e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40434/0 (socket says 172.21.15.83:40434) 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 1 -- 172.21.15.83:0/2214538630 learned_addr learned my addr 172.21.15.83:0/2214538630 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 10 monclient(hunting): get_auth_request con 0x7f656c140b10 auth_method 0 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656affd640 10 monclient(hunting): get_auth_request con 0x7f656c0b2df0 auth_method 0 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:34.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656b7fe640 10 monclient(hunting): get_auth_request con 0x7f656c0b04a0 auth_method 0 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.582+0000 7f656affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.583+0000 7f656b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.583+0000 7f656b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.583+0000 7f656b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:34.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2024-03-25T02:00:34.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 1 -- 172.21.15.83:0/2214538630 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 msgr2=0x7f656c1340b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c1340b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 1 -- 172.21.15.83:0/2214538630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c0b2df0 msgr2=0x7f656c12dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c0b2df0 0x7f656c12dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 1 monclient: found mon.noname-b 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.585+0000 7f656bfff640 1 -- 172.21.15.83:0/2214538630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c143ba0 con 0x7f656c140b10 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.589+0000 7f656bfff640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f656c140b10 0x7f656c12e1d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f655c02c540 tx=0x7f655c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.589+0000 7f6563fff640 10 client.?.objecter ms_handle_connect 0x7f656c140b10 2024-03-25T02:00:34.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.589+0000 7f6563fff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:34.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 1 -- 172.21.15.83:0/2214538630 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c044020 con 0x7f656c140b10 2024-03-25T02:00:34.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:34.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:34.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: dump: 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: _finish_auth 0 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 1 -- 172.21.15.83:0/2214538630 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655c03f470 con 0x7f656c140b10 2024-03-25T02:00:34.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 1 -- 172.21.15.83:0/2214538630 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c039040 con 0x7f656c140b10 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 5 monclient: authenticate success, global_id 4306 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 10 monclient: _renew_subs 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c144080 con 0x7f656c140b10 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6569ffb640 4 set_mon_vals no callback set 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6563fff640 10 monclient: dump: 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:34.597 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 10 monclient: _renew_subs 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c137ab0 con 0x7f656c140b10 2024-03-25T02:00:34.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.593+0000 7f6571a7b640 1 librados: init done 2024-03-25T02:00:34.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.594+0000 7f6571a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:34.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.594+0000 7f6571a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:34.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.594+0000 7f6571a7b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:34.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.594+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f656c075810 con 0x7f656c140b10 2024-03-25T02:00:34.848 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.846+0000 7f6563fff640 1 -- 172.21.15.83:0/2214538630 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f655c047020 con 0x7f656c140b10 2024-03-25T02:00:34.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.846+0000 7f6563fff640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f654c06da20 0x7f654c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:34.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.847+0000 7f656b7fe640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f654c06da20 0x7f654c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:34.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.847+0000 7f656b7fe640 10 monclient: get_auth_request con 0x7f654c06da20 auth_method 0 2024-03-25T02:00:34.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.847+0000 7f6563fff640 10 client.4306.objecter ms_handle_connect 0x7f654c06da20 2024-03-25T02:00:34.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.847+0000 7f656b7fe640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f654c06da20 0x7f654c06fee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6564009aa0 tx=0x7f6564009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:34.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.849+0000 7f6563fff640 1 -- 172.21.15.83:0/2214538630 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5210+0+0 (secure 0 0 0) 0x7f655c07a4c0 con 0x7f656c140b10 2024-03-25T02:00:34.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.850+0000 7f6563fff640 10 client.4306.objecter ms_dispatch 0x7f656c000b90 osd_map(24..24 src has 1..24) 2024-03-25T02:00:34.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.850+0000 7f6563fff640 3 client.4306.objecter handle_osd_map got epochs [24,24] > 0 2024-03-25T02:00:34.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.850+0000 7f6563fff640 3 client.4306.objecter handle_osd_map decoding full epoch 24 2024-03-25T02:00:34.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.850+0000 7f6563fff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T02:00:34.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.860+0000 7f6563fff640 1 -- 172.21.15.83:0/2214538630 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f655c04e020 con 0x7f656c140b10 2024-03-25T02:00:34.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.860+0000 7f6563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:34.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:34.860+0000 7f6563fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:35.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.068+0000 7f6571a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:35.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.068+0000 7f6571a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:35.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.068+0000 7f6571a7b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:35.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.068+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f656c143d70 con 0x7f656c140b10 2024-03-25T02:00:35.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.102+0000 7f6563fff640 1 -- 172.21.15.83:0/2214538630 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+14742 (secure 0 0 0) 0x7f655c07c420 con 0x7f656c140b10 2024-03-25T02:00:35.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.102+0000 7f6563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:35.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.102+0000 7f6563fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:35.104 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:00:35.104 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":24,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T02:00:34.060321+0000","last_up_change":"2024-03-25T02:00:23.723108+0000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-25T02:00:14.575846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-25T02:00:31.420756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6801","nonce":2678950219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6803","nonce":2678950219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6807","nonce":2678950219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6805","nonce":2678950219}]},"public_addr":"172.21.15.83:6801/2678950219","cluster_addr":"172.21.15.83:6803/2678950219","heartbeat_back_addr":"172.21.15.83:6807/2678950219","heartbeat_front_addr":"172.21.15.83:6805/2678950219","state":["exists","up"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6825","nonce":2597989034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6827","nonce":2597989034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6831","nonce":2597989034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6829","nonce":2597989034}]},"public_addr":"172.21.15.83:6825/2597989034","cluster_addr":"172.21.15.83:6827/2597989034","heartbeat_back_addr":"172.21.15.83:6831/2597989034","heartbeat_front_addr":"172.21.15.83:6829/2597989034","state":["exists","up"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6817","nonce":602002618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6819","nonce":602002618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6823","nonce":602002618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6821","nonce":602002618}]},"public_addr":"172.21.15.83:6817/602002618","cluster_addr":"172.21.15.83:6819/602002618","heartbeat_back_addr":"172.21.15.83:6823/602002618","heartbeat_front_addr":"172.21.15.83:6821/602002618","state":["exists","up"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6809","nonce":3432501411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6811","nonce":3432501411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6815","nonce":3432501411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6813","nonce":3432501411}]},"public_addr":"172.21.15.83:6809/3432501411","cluster_addr":"172.21.15.83:6811/3432501411","heartbeat_back_addr":"172.21.15.83:6815/3432501411","heartbeat_front_addr":"172.21.15.83:6813/3432501411","state":["exists","up"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6819","nonce":2579144563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6821","nonce":2579144563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6825","nonce":2579144563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6823","nonce":2579144563}]},"public_addr":"172.21.15.153:6819/2579144563","cluster_addr":"172.21.15.153:6821/2579144563","heartbeat_back_addr":"172.21.15.153:6825/2579144563","heartbeat_front_addr":"172.21.15.153:6823/2579144563","state":["exists","up"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6811","nonce":2458137144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6813","nonce":2458137144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6817","nonce":2458137144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6815","nonce":2458137144}]},"public_addr":"172.21.15.153:6811/2458137144","cluster_addr":"172.21.15.153:6813/2458137144","heartbeat_back_addr":"172.21.15.153:6817/2458137144","heartbeat_front_addr":"172.21.15.153:6815/2458137144","state":["exists","up"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6803","nonce":4205760170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6805","nonce":4205760170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6809","nonce":4205760170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6807","nonce":4205760170}]},"public_addr":"172.21.15.153:6803/4205760170","cluster_addr":"172.21.15.153:6805/4205760170","heartbeat_back_addr":"172.21.15.153:6809/4205760170","heartbeat_front_addr":"172.21.15.153:6807/4205760170","state":["exists","up"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6827","nonce":2580901001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6829","nonce":2580901001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6833","nonce":2580901001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6831","nonce":2580901001}]},"public_addr":"172.21.15.153:6827/2580901001","cluster_addr":"172.21.15.153:6829/2580901001","heartbeat_back_addr":"172.21.15.153:6833/2580901001","heartbeat_front_addr":"172.21.15.153:6831/2580901001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.217280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.797510+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.322329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.630439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.336792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:09.768567+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.625955+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.445663+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 10 librados: watch_flush enter 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 10 librados: watch_flush exit 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 10 client.4306.objecter successfully canceled tick 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f654c06da20 msgr2=0x7f654c06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f654c06da20 0x7f654c06fee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6564009aa0 tx=0x7f6564009190 comp rx=0 tx=0).stop 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 10 monclient: shutdown 2024-03-25T02:00:35.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f656c140b10 msgr2=0x7f656c12e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f656c140b10 0x7f656c12e1d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f655c02c540 tx=0x7f655c004a40 comp rx=0 tx=0).stop 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 shutdown_connections 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f654c06da20 0x7f654c06fee0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f656c140b10 0x7f656c12e1d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f656c0b2df0 0x7f656c12dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 --2- 172.21.15.83:0/2214538630 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f656c0b04a0 0x7f656c1340b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 >> 172.21.15.83:0/2214538630 conn(0x7f656c0b8a00 msgr2=0x7f656c0b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 shutdown_connections 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 -- 172.21.15.83:0/2214538630 wait complete. 2024-03-25T02:00:35.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.104+0000 7f6571a7b640 1 librados: shutdown 2024-03-25T02:00:35.118 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-25T02:00:35.118 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.237+0000 7f37b9a8f640 10 monclient: get_monmap_and_config 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.237+0000 7f37b9a8f640 10 monclient: build_initial_monmap 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.237+0000 7f37b9a8f640 10 monclient: monmap: 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:35.239092+0000 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:35.239092+0000 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:35.240 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:35.241 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:35.241 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:35.241 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:35.241 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:35.241 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:35.241 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:35.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.238+0000 7f37b9a8f640 1 Processor -- start 2024-03-25T02:00:35.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.238+0000 7f37b9a8f640 1 -- start start 2024-03-25T02:00:35.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.238+0000 7f37b9a8f640 10 monclient: init 2024-03-25T02:00:35.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37b4067500 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b4146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f37b4140a10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f37b4146ed0 0x7f37b4147340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f37b4146ed0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37b40af5c0 con 0x7f37b4067500 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f37b40678e0 con 0x7f37b4146ed0 2024-03-25T02:00:35.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:35.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f37b4140e50 con 0x7f37b4140a10 2024-03-25T02:00:35.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:35.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.239+0000 7f37b9a8f640 10 monclient(hunting): authenticate will time out at 1354.429798s 2024-03-25T02:00:35.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.274+0000 7f37b2ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b4146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:35.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.276+0000 7f37b2ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b4146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54372/0 (socket says 172.21.15.83:54372) 2024-03-25T02:00:35.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.276+0000 7f37b2ffd640 1 -- 172.21.15.83:0/2930751937 learned_addr learned my addr 172.21.15.83:0/2930751937 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:35.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.276+0000 7f37b2ffd640 10 monclient(hunting): get_auth_request con 0x7f37b4140a10 auth_method 0 2024-03-25T02:00:35.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.276+0000 7f37b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:35.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.276+0000 7f37b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:35.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.276+0000 7f37b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:35.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.277+0000 7f37b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:35.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.277+0000 7f37b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:35.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.277+0000 7f37b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:35.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-03-25T02:00:35.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 1 -- 172.21.15.83:0/2930751937 >> v2:172.21.15.153:3300/0 conn(0x7f37b4146ed0 msgr2=0x7f37b4147340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:35.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 1 --2- 172.21.15.83:0/2930751937 >> v2:172.21.15.153:3300/0 conn(0x7f37b4146ed0 0x7f37b4147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 1 -- 172.21.15.83:0/2930751937 >> v1:172.21.15.153:6789/0 conn(0x7f37b4067500 legacy=0x7f37b4140300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:35.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:00:35.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 1 monclient: found mon.noname-b 2024-03-25T02:00:35.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:35.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:35.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.279+0000 7f37b2ffd640 1 -- 172.21.15.83:0/2930751937 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b406fe30 con 0x7f37b4140a10 2024-03-25T02:00:35.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.283+0000 7f37b2ffd640 1 --2- 172.21.15.83:0/2930751937 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b4146990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009870 tx=0x7f37a802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37be27faeb6dbc08 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 1 -- 172.21.15.83:0/2930751937 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a8035020 con 0x7f37b4140a10 2024-03-25T02:00:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: dump: 2024-03-25T02:00:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:35.289 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 1 -- 172.21.15.83:0/2930751937 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a8031640 con 0x7f37b4140a10 2024-03-25T02:00:35.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 1 -- 172.21.15.83:0/2930751937 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a802d040 con 0x7f37b4140a10 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b9a8f640 5 monclient: authenticate success, global_id 4232 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b9a8f640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b9a8f640 4 set_mon_vals no callback set 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.287+0000 7f37b27fc640 10 monclient: dump: 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.291 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:35.292 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:35.292 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:35.292 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:35.292 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:35.292 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:35.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 10 monclient: shutdown 2024-03-25T02:00:35.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:35.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 -- 172.21.15.83:0/2930751937 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 msgr2=0x7f37b4146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/2930751937 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b4146990 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009870 tx=0x7f37a802b8f0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 -- 172.21.15.83:0/2930751937 shutdown_connections 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/2930751937 >> v2:172.21.15.153:3300/0 conn(0x7f37b4146ed0 0x7f37b4147340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/2930751937 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b4146990 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 -- 172.21.15.83:0/2930751937 >> 172.21.15.83:0/2930751937 conn(0x7f37b40b5550 msgr2=0x7f37b40b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 -- 172.21.15.83:0/2930751937 shutdown_connections 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.288+0000 7f37b9a8f640 1 -- 172.21.15.83:0/2930751937 wait complete. 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 10 monclient: build_initial_monmap 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 10 monclient: monmap: 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:35.296 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 1 librados: starting msgr at 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 1 librados: starting objecter 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 1 Processor -- start 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 1 -- start start 2024-03-25T02:00:35.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 1 librados: setting wanted keys 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 1 librados: calling monclient init 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.289+0000 7f37b9a8f640 10 monclient: init 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37b4067500 0x7f37b4141e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f37b4067500 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b413dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f37b4140a10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37b4146ed0 0x7f37b413e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:35.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f37b4146ed0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37b40ade10 con 0x7f37b4067500 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37b40ae220 con 0x7f37b4146ed0 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f37b40adf90 con 0x7f37b4140a10 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b9a8f640 10 monclient(hunting): authenticate will time out at 1354.480426s 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37b4067500 0x7f37b4141e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.290+0000 7f37b3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37b4146ed0 0x7f37b413e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b2ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b413dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37b4146ed0 0x7f37b413e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40460/0 (socket says 172.21.15.83:40460) 2024-03-25T02:00:35.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 1 -- 172.21.15.83:0/636778070 learned_addr learned my addr 172.21.15.83:0/636778070 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 10 monclient(hunting): get_auth_request con 0x7f37b4146ed0 auth_method 0 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b37fe640 10 monclient(hunting): get_auth_request con 0x7f37b4067500 auth_method 0 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b2ffd640 10 monclient(hunting): get_auth_request con 0x7f37b4140a10 auth_method 0 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:35.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.291+0000 7f37b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.292+0000 7f37b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.292+0000 7f37b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.292+0000 7f37b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.292+0000 7f37b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.292+0000 7f37b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:35.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.292+0000 7f37b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:35.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 10 monclient(hunting): handle_auth_done global_id 4309 payload 824 2024-03-25T02:00:35.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 1 -- 172.21.15.83:0/636778070 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 msgr2=0x7f37b413dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:35.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b413dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 1 -- 172.21.15.83:0/636778070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37b4067500 msgr2=0x7f37b4141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:35.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37b4067500 0x7f37b4141e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 1 monclient: found mon.noname-b 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.294+0000 7f37b3fff640 1 -- 172.21.15.83:0/636778070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b4143fe0 con 0x7f37b4146ed0 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.296+0000 7f37b3fff640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37b4146ed0 0x7f37b413e1c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00b970 tx=0x7f37ac00be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.296+0000 7f379bfff640 10 client.?.objecter ms_handle_connect 0x7f37b4146ed0 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.296+0000 7f379bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f379bfff640 1 -- 172.21.15.83:0/636778070 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac019070 con 0x7f37b4146ed0 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f379bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f379bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f379bfff640 10 monclient: dump: 2024-03-25T02:00:35.304 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f379bfff640 10 monclient: _finish_auth 0 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f379bfff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f379bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:35.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 5 monclient: authenticate success, global_id 4309 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 10 monclient: _renew_subs 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b4144250 con 0x7f37b4146ed0 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 10 monclient: _renew_subs 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b4140fd0 con 0x7f37b4146ed0 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.298+0000 7f37b9a8f640 1 librados: init done 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.299+0000 7f37b9a8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.299+0000 7f37b9a8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.299+0000 7f37b9a8f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:35.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.299+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b4075810 con 0x7f37b4146ed0 2024-03-25T02:00:35.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.505+0000 7f379bfff640 1 -- 172.21.15.83:0/636778070 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ac00cb40 con 0x7f37b4146ed0 2024-03-25T02:00:35.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.505+0000 7f379bfff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:35.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.505+0000 7f37b1ffb640 4 set_mon_vals no callback set 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.545+0000 7f379bfff640 1 -- 172.21.15.83:0/636778070 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac023060 con 0x7f37b4146ed0 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.545+0000 7f379bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.545+0000 7f379bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.545+0000 7f379bfff640 10 monclient: dump: 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:35.547 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:35.548 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:35.548 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:35.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f379bfff640 1 -- 172.21.15.83:0/636778070 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f37ac01c070 con 0x7f37b4146ed0 2024-03-25T02:00:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f379bfff640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f37a006da20 0x7f37a006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f379bfff640 1 -- 172.21.15.83:0/636778070 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5210+0+0 (secure 0 0 0) 0x7f37ac08b300 con 0x7f37b4146ed0 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f379bfff640 10 client.4309.objecter ms_dispatch 0x7f37b4000b90 osd_map(24..24 src has 1..24) 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f379bfff640 3 client.4309.objecter handle_osd_map got epochs [24,24] > 0 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f379bfff640 3 client.4309.objecter handle_osd_map decoding full epoch 24 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f379bfff640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f37b37fe640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f37a006da20 0x7f37a006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.559+0000 7f37b37fe640 10 monclient: get_auth_request con 0x7f37a006da20 auth_method 0 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.560+0000 7f379bfff640 10 client.4309.objecter ms_handle_connect 0x7f37a006da20 2024-03-25T02:00:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.560+0000 7f37b37fe640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f37a006da20 0x7f37a006fee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37a40017f0 tx=0x7f37a4009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:35.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.578+0000 7f379bfff640 1 -- 172.21.15.83:0/636778070 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f37ac0597b0 con 0x7f37b4146ed0 2024-03-25T02:00:35.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.578+0000 7f379bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:35.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.578+0000 7f379bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:35.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.798+0000 7f37b9a8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.798+0000 7f37b9a8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.798+0000 7f37b9a8f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:35.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.798+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37b4147340 con 0x7f37b4146ed0 2024-03-25T02:00:35.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.826+0000 7f379bfff640 1 -- 172.21.15.83:0/636778070 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+14729 (secure 0 0 0) 0x7f37ac05d850 con 0x7f37b4146ed0 2024-03-25T02:00:35.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.826+0000 7f379bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:00:35.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.826+0000 7f379bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:35.828 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:00:35.828 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":25,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T02:00:35.418213+0000","last_up_change":"2024-03-25T02:00:23.723108+0000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-25T02:00:14.575846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-25T02:00:31.420756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6801","nonce":2678950219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6803","nonce":2678950219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6807","nonce":2678950219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6805","nonce":2678950219}]},"public_addr":"172.21.15.83:6801/2678950219","cluster_addr":"172.21.15.83:6803/2678950219","heartbeat_back_addr":"172.21.15.83:6807/2678950219","heartbeat_front_addr":"172.21.15.83:6805/2678950219","state":["exists","up"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6825","nonce":2597989034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6827","nonce":2597989034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6831","nonce":2597989034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6829","nonce":2597989034}]},"public_addr":"172.21.15.83:6825/2597989034","cluster_addr":"172.21.15.83:6827/2597989034","heartbeat_back_addr":"172.21.15.83:6831/2597989034","heartbeat_front_addr":"172.21.15.83:6829/2597989034","state":["exists","up"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6817","nonce":602002618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6819","nonce":602002618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6823","nonce":602002618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6821","nonce":602002618}]},"public_addr":"172.21.15.83:6817/602002618","cluster_addr":"172.21.15.83:6819/602002618","heartbeat_back_addr":"172.21.15.83:6823/602002618","heartbeat_front_addr":"172.21.15.83:6821/602002618","state":["exists","up"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6809","nonce":3432501411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6811","nonce":3432501411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6815","nonce":3432501411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6813","nonce":3432501411}]},"public_addr":"172.21.15.83:6809/3432501411","cluster_addr":"172.21.15.83:6811/3432501411","heartbeat_back_addr":"172.21.15.83:6815/3432501411","heartbeat_front_addr":"172.21.15.83:6813/3432501411","state":["exists","up"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6819","nonce":2579144563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6821","nonce":2579144563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6825","nonce":2579144563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6823","nonce":2579144563}]},"public_addr":"172.21.15.153:6819/2579144563","cluster_addr":"172.21.15.153:6821/2579144563","heartbeat_back_addr":"172.21.15.153:6825/2579144563","heartbeat_front_addr":"172.21.15.153:6823/2579144563","state":["exists","up"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6811","nonce":2458137144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6813","nonce":2458137144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6817","nonce":2458137144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6815","nonce":2458137144}]},"public_addr":"172.21.15.153:6811/2458137144","cluster_addr":"172.21.15.153:6813/2458137144","heartbeat_back_addr":"172.21.15.153:6817/2458137144","heartbeat_front_addr":"172.21.15.153:6815/2458137144","state":["exists","up"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6803","nonce":4205760170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6805","nonce":4205760170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6809","nonce":4205760170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6807","nonce":4205760170}]},"public_addr":"172.21.15.153:6803/4205760170","cluster_addr":"172.21.15.153:6805/4205760170","heartbeat_back_addr":"172.21.15.153:6809/4205760170","heartbeat_front_addr":"172.21.15.153:6807/4205760170","state":["exists","up"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6827","nonce":2580901001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6829","nonce":2580901001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6833","nonce":2580901001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6831","nonce":2580901001}]},"public_addr":"172.21.15.153:6827/2580901001","cluster_addr":"172.21.15.153:6829/2580901001","heartbeat_back_addr":"172.21.15.153:6833/2580901001","heartbeat_front_addr":"172.21.15.153:6831/2580901001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.217280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.797510+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.322329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.630439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.336792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:09.768567+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.625955+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.445663+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 10 librados: watch_flush enter 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 10 librados: watch_flush exit 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 10 client.4309.objecter successfully canceled tick 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f37a006da20 msgr2=0x7f37a006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f37a006da20 0x7f37a006fee0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f37a40017f0 tx=0x7f37a4009040 comp rx=0 tx=0).stop 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 10 monclient: shutdown 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37b4146ed0 msgr2=0x7f37b413e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37b4146ed0 0x7f37b413e1c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00b970 tx=0x7f37ac00be30 comp rx=0 tx=0).stop 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 shutdown_connections 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f37a006da20 0x7f37a006fee0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37b4146ed0 0x7f37b413e1c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37b4140a10 0x7f37b413dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 --2- 172.21.15.83:0/636778070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37b4067500 0x7f37b4141e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 >> 172.21.15.83:0/636778070 conn(0x7f37b40b5550 msgr2=0x7f37b41400d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 shutdown_connections 2024-03-25T02:00:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 -- 172.21.15.83:0/636778070 wait complete. 2024-03-25T02:00:35.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:35.828+0000 7f37b9a8f640 1 librados: shutdown 2024-03-25T02:00:35.842 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-25T02:00:35.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-25T02:00:35.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-25T02:00:35.843 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-25T02:00:35.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-25T02:00:35.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-25T02:00:35.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-25T02:00:35.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-25T02:00:36.014 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.012+0000 7f1ffd825640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.012+0000 7f1ffd825640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.012+0000 7f1ffd825640 10 monclient: monmap: 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.014069+0000 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.014069+0000 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.015 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7fe67cdaf640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7fe67cdaf640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7fe67cdaf640 10 monclient: monmap: 2024-03-25T02:00:36.016 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.015539+0000 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.015539+0000 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.017 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.013+0000 7f1ffd825640 1 Processor -- start 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.013+0000 7f1ffd825640 1 -- start start 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.013+0000 7f1ffd825640 10 monclient: init 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ff0039010 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ff0035670 0x7f1ff00cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.018 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff0035670 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff00cc630 0x7f1ff00d6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff00cc630 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ff0037410 con 0x7f1ff0039010 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f1ff00393f0 con 0x7f1ff0035670 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff0035c20 con 0x7f1ff00cc630 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.014+0000 7f1ffd825640 10 monclient(hunting): authenticate will time out at 1355.204617s 2024-03-25T02:00:36.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.015+0000 7fe67cdaf640 1 Processor -- start 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.015+0000 7f975b4e4640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.015+0000 7f975b4e4640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.015+0000 7f975b4e4640 10 monclient: monmap: 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.016728+0000 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.016728+0000 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.020 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.015+0000 7fe67cdaf640 1 -- start start 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.015+0000 7fe67cdaf640 10 monclient: init 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe6700c1080 0x7fe6700c1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe6700c1080 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe6700069f0 0x7fe6700349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe6700069f0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe670031fb0 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6700317f0 con 0x7fe670031fb0 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe670031a90 con 0x7fe6700069f0 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fe6700291e0 con 0x7fe6700c1080 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7fe67cdaf640 10 monclient(hunting): authenticate will time out at 1355.206329s 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f1ff67fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ff0035670 0x7f1ff00cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f1ff67fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f1ff0035670 0x7f1ff00cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55872/0 (socket says 172.21.15.83:55872) 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f1ff67fc640 1 -- 172.21.15.83:0/4244739934 learned_addr learned my addr 172.21.15.83:0/4244739934 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f1ff67fc640 10 monclient(hunting): get_auth_request con 0x7f1ff0035670 auth_method 0 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f1ff67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f1ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f1ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f2c2da45640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f2c2da45640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f2c2da45640 10 monclient: monmap: 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.017450+0000 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.017450+0000 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.024 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.025 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.025 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.025 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.025 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe6700c1080 0x7fe6700c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f2c2da45640 1 Processor -- start 2024-03-25T02:00:36.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe6700c1080 0x7fe6700c1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40462/0 (socket says 172.21.15.83:40462) 2024-03-25T02:00:36.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 1 -- 172.21.15.83:0/1645066816 learned_addr learned my addr 172.21.15.83:0/1645066816 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 10 monclient(hunting): get_auth_request con 0x7fe6700c1080 auth_method 0 2024-03-25T02:00:36.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7fe67659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f2c2da45640 1 -- start start 2024-03-25T02:00:36.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f2c2da45640 10 monclient: init 2024-03-25T02:00:36.029 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff77fe640 1 --2- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff00cc630 0x7f1ff00d6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff77fe640 10 monclient(hunting): get_auth_request con 0x7f1ff00cc630 auth_method 0 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f1ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f1ff77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f1ff77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f1ff77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c200c10c0 0x7f2c200c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient(hunting): picked mon.noname-b con 0x7f2c200c10c0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2c20006a30 0x7f2c200349e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient(hunting): picked mon.noname-c con 0x7f2c20006a30 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c20031ff0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c20031830 con 0x7f2c20031ff0 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f2c20031ad0 con 0x7f2c20006a30 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2c20029220 con 0x7f2c200c10c0 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f2c2da45640 10 monclient(hunting): authenticate will time out at 1355.208525s 2024-03-25T02:00:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.016+0000 7f975b4e4640 1 Processor -- start 2024-03-25T02:00:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f975b4e4640 1 -- start start 2024-03-25T02:00:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.017+0000 7f975b4e4640 10 monclient: init 2024-03-25T02:00:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-03-25T02:00:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f975406b870 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9754070630 0x7f97540720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient(hunting): picked mon.noname-b con 0x7f9754070630 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97541446b0 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97540719c0 con 0x7f97541446b0 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9754071ca0 con 0x7f975406b870 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f975406bc50 con 0x7f9754070630 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.018+0000 7f975b4e4640 10 monclient(hunting): authenticate will time out at 1355.208122s 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-03-25T02:00:36.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 1 -- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff00cc630 msgr2=0x7f1ff00d6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 1 --2- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff00cc630 0x7f1ff00d6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 1 -- 172.21.15.83:0/4244739934 >> v1:172.21.15.83:6789/0 conn(0x7f1ff0039010 legacy=0x7f1ff00c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 1 monclient: found mon.noname-a 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f1ff67fc640 1 -- 172.21.15.83:0/4244739934 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff0037680 con 0x7f1ff0035670 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.020+0000 7f65bef99640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.020+0000 7f65bef99640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.020+0000 7f65bef99640 10 monclient: monmap: 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.021850+0000 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.021850+0000 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.035 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 1 -- 172.21.15.83:0/1645066816 >> v2:172.21.15.83:3300/0 conn(0x7fe6700069f0 msgr2=0x7fe6700349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 1 --2- 172.21.15.83:0/1645066816 >> v2:172.21.15.83:3300/0 conn(0x7fe6700069f0 0x7fe6700349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 1 -- 172.21.15.83:0/1645066816 >> v1:172.21.15.83:6789/0 conn(0x7fe670031fb0 legacy=0x7fe6700c3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7fe67659c640 1 -- 172.21.15.83:0/1645066816 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6700333b0 con 0x7fe6700c1080 2024-03-25T02:00:36.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.020+0000 7f2c267fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2c20006a30 0x7f2c200349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f8eb5792640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f65bef99640 1 Processor -- start 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f8eb5792640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.019+0000 7f8eb5792640 10 monclient: monmap: 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.021105+0000 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.021105+0000 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.038 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f2c267fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2c20006a30 0x7f2c200349e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40478/0 (socket says 172.21.15.83:40478) 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f2c267fc640 1 -- 172.21.15.83:0/3892219497 learned_addr learned my addr 172.21.15.83:0/3892219497 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f2c267fc640 10 monclient(hunting): get_auth_request con 0x7f2c20006a30 auth_method 0 2024-03-25T02:00:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f2c267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f2c267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f2c267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7fc7e7fff640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7fc7e7fff640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f8eb5792640 1 Processor -- start 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7fc7e7fff640 10 monclient: monmap: 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.023149+0000 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.023149+0000 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.041 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.042 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.042 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.042 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.042 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f65bef99640 1 -- start start 2024-03-25T02:00:36.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f65bef99640 10 monclient: init 2024-03-25T02:00:36.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7fc7e7fff640 1 Processor -- start 2024-03-25T02:00:36.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7fc7e7fff640 1 -- start start 2024-03-25T02:00:36.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7fc7e7fff640 10 monclient: init 2024-03-25T02:00:36.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f8eb5792640 1 -- start start 2024-03-25T02:00:36.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.021+0000 7f8eb5792640 10 monclient: init 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65b0034c20 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b002fc10 0x7f65b00c9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient(hunting): picked mon.noname-b con 0x7f65b002fc10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65b00cf5c0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b0031d10 con 0x7f65b0034c20 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b0032f80 con 0x7f65b00cf5c0 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f65b00321f0 con 0x7f65b002fc10 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.044 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f65bef99640 10 monclient(hunting): authenticate will time out at 1355.212750s 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f8eb5792640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f8eb5792640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f8eb5792640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8eb0072e50 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f8eb5792640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f8eb0073230 0x7f8eb00737d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f8eb5792640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb0073230 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f8eb5792640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8eb00aeca0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f2c267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f2c267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f2c267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f9759280640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f975406b870 0x7f97540b15b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33266/0 (socket says 172.21.15.83:33266) 2024-03-25T02:00:36.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f9759280640 1 -- 172.21.15.83:0/4199886810 learned_addr learned my addr 172.21.15.83:0/4199886810 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.022+0000 7f8eb5792640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb0142be0 con 0x7f8eb0072e50 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f8eb5792640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb0072930 con 0x7f8eb00aeca0 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f8eb5792640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f8eb5792640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f8eb0070aa0 con 0x7f8eb0073230 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f8eb5792640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f8eb5792640 10 monclient(hunting): authenticate will time out at 1355.213029s 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-03-25T02:00:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f1ff67fc640 1 --2- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0035670 0x7f1ff00cc0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0035da0 tx=0x7f1fe802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6529a2ed8c6a512a server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e0038da0 0x7fc7e00c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc7e0038da0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.048 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fc7e0039180 0x7fc7e00c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f998d359640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f998d359640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7f998d359640 10 monclient: monmap: 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.024841+0000 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.024841+0000 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.049 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.050 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc7e0039180 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7e00c5790 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7e00c7e90 con 0x7fc7e00c5790 2024-03-25T02:00:36.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fc7e0035460 con 0x7fc7e0039180 2024-03-25T02:00:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fc7e0035890 con 0x7fc7e0038da0 2024-03-25T02:00:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.023+0000 7fc7e7fff640 10 monclient(hunting): authenticate will time out at 1355.213524s 2024-03-25T02:00:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 1 Processor -- start 2024-03-25T02:00:36.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 1 -- start start 2024-03-25T02:00:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 10 monclient: init 2024-03-25T02:00:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-03-25T02:00:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 1 -- 172.21.15.83:0/3892219497 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c200c10c0 msgr2=0x7f2c200c14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 1 --2- 172.21.15.83:0/3892219497 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c200c10c0 0x7f2c200c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 1 -- 172.21.15.83:0/3892219497 >> v1:172.21.15.153:6789/0 conn(0x7f2c20031ff0 legacy=0x7f2c200c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.052 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f9758a7f640 1 --2- 172.21.15.83:0/4199886810 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9754070630 0x7f97540720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.053 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f2c267fc640 1 -- 172.21.15.83:0/3892219497 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c200333f0 con 0x7f2c20006a30 2024-03-25T02:00:36.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9988067500 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f99880737f0 0x7f99881430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 10 monclient(hunting): picked mon.noname-b con 0x7f99880737f0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.024+0000 7f998d359640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f998813cef0 0x7f998813d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 10 monclient(hunting): picked mon.noname-a con 0x7f998813cef0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9988073c10 con 0x7f9988067500 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f99880678e0 con 0x7f998813cef0 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.055 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f99880aeed0 con 0x7f99880737f0 2024-03-25T02:00:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f998d359640 10 monclient(hunting): authenticate will time out at 1355.215092s 2024-03-25T02:00:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f9758a7f640 10 monclient(hunting): get_auth_request con 0x7f9754070630 auth_method 0 2024-03-25T02:00:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f9758a7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f9758a7f640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.056 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.025+0000 7f9758a7f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.026+0000 7f9758a7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.026+0000 7f9758a7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.026+0000 7f9758a7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.027+0000 7fe67659c640 1 --2- 172.21.15.83:0/1645066816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6700c1080 0x7fe6700c1460 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe67806a5d0 tx=0x7fe6780a7190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd2f0b704277b5dc server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.057 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.027+0000 7f2c267fc640 1 --2- 172.21.15.83:0/3892219497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c20006a30 0x7f2c200349e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e14ccd0b6d4430d server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-03-25T02:00:36.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 1 -- 172.21.15.83:0/4199886810 >> v1:172.21.15.153:6789/0 conn(0x7f975406b870 legacy=0x7f97540b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 1 -- 172.21.15.83:0/4199886810 >> v1:172.21.15.83:6789/0 conn(0x7f97541446b0 legacy=0x7f9754151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7fc7e6ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e0038da0 0x7fc7e00c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.028+0000 7f9758a7f640 1 -- 172.21.15.83:0/4199886810 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97540b1cc0 con 0x7f9754070630 2024-03-25T02:00:36.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.029+0000 7fc7e6ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e0038da0 0x7fc7e00c5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54432/0 (socket says 172.21.15.83:54432) 2024-03-25T02:00:36.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.029+0000 7fc7e6ffd640 1 -- 172.21.15.83:0/2151176330 learned_addr learned my addr 172.21.15.83:0/2151176330 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.029+0000 7fc7e6ffd640 10 monclient(hunting): get_auth_request con 0x7fc7e0038da0 auth_method 0 2024-03-25T02:00:36.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.029+0000 7fc7e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.029+0000 7fc7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.029+0000 7fc7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.060 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.029+0000 7f65bd796640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b002fc10 0x7f65b00c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 1 -- 172.21.15.83:0/4244739934 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80317c0 con 0x7f1ff0035670 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: dump: 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.062 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 1 -- 172.21.15.83:0/4244739934 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe8037070 con 0x7f1ff0035670 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 1 -- 172.21.15.83:0/4244739934 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8031b80 con 0x7f1ff0035670 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ffd825640 5 monclient: authenticate success, global_id 4296 2024-03-25T02:00:36.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ffd825640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ffd825640 4 set_mon_vals no callback set 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.030+0000 7f1ff5ffb640 10 monclient: dump: 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 10 monclient: shutdown 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 -- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0035670 msgr2=0x7f1ff00cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 --2- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0035670 0x7f1ff00cc0f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0035da0 tx=0x7f1fe802b860 comp rx=0 tx=0).stop 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 -- 172.21.15.83:0/4244739934 shutdown_connections 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 --2- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff00cc630 0x7f1ff00d6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 --2- 172.21.15.83:0/4244739934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0035670 0x7f1ff00cc0f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 -- 172.21.15.83:0/4244739934 >> 172.21.15.83:0/4244739934 conn(0x7f1ff0006600 msgr2=0x7f1ff0006a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 -- 172.21.15.83:0/4244739934 shutdown_connections 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7f1ffd825640 1 -- 172.21.15.83:0/4244739934 wait complete. 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.031+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 10 monclient: monmap: 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.066 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f65bd796640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b002fc10 0x7f65b00c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54416/0 (socket says 172.21.15.83:54416) 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f65bd796640 1 -- 172.21.15.83:0/849601417 learned_addr learned my addr 172.21.15.83:0/849601417 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f65bd796640 10 monclient(hunting): get_auth_request con 0x7f65b002fc10 auth_method 0 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f65bd796640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f65bd796640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.067 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f65bd796640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 1 librados: starting msgr at 2024-03-25T02:00:36.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 1 librados: starting objecter 2024-03-25T02:00:36.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 1 Processor -- start 2024-03-25T02:00:36.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 1 -- start start 2024-03-25T02:00:36.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 1 librados: setting wanted keys 2024-03-25T02:00:36.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 1 librados: calling monclient init 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.032+0000 7f1ffd825640 10 monclient: init 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff0035670 0x7f1ff00c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff0035670 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0039010 0x7f1ff00c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff0039010 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ff00cc630 0x7f1ff00c0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff00cc630 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff00ccc20 con 0x7f1ff0039010 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff00ccaa0 con 0x7f1ff00cc630 2024-03-25T02:00:36.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff00ccda0 con 0x7f1ff0035670 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f1ffd825640 10 monclient(hunting): authenticate will time out at 1355.223196s 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 1 -- 172.21.15.83:0/2151176330 >> v2:172.21.15.83:3300/0 conn(0x7fc7e0039180 msgr2=0x7fc7e00c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 1 --2- 172.21.15.83:0/2151176330 >> v2:172.21.15.83:3300/0 conn(0x7fc7e0039180 0x7fc7e00c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 1 -- 172.21.15.83:0/2151176330 >> v1:172.21.15.153:6789/0 conn(0x7fc7e00c5790 legacy=0x7fc7e00cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7fc7e6ffd640 1 -- 172.21.15.83:0/2151176330 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e0036090 con 0x7fc7e0038da0 2024-03-25T02:00:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.033+0000 7f9758a7f640 1 --2- 172.21.15.83:0/4199886810 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9754070630 0x7f97540720c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97440097d0 tx=0x7f974402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d9a6085ee5176 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.034+0000 7f9986ffd640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f9988067500 0x7f9988071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45644/0 (socket says 172.21.15.83:45644) 2024-03-25T02:00:36.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.034+0000 7f9986ffd640 1 -- 172.21.15.83:0/1008587590 learned_addr learned my addr 172.21.15.83:0/1008587590 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.035+0000 7f65bd796640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.035+0000 7f65bd796640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.035+0000 7f65bd796640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.035+0000 7f8eaffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f8eb0072e50 0x7f8eb013c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45640/0 (socket says 172.21.15.83:45640) 2024-03-25T02:00:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.035+0000 7f8eaffff640 1 -- 172.21.15.83:0/3114093314 learned_addr learned my addr 172.21.15.83:0/3114093314 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:36.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7f9987fff640 1 --2- 172.21.15.83:0/1008587590 >> v2:172.21.15.83:3300/0 conn(0x7f998813cef0 0x7f998813d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 1 -- 172.21.15.83:0/1645066816 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6780b4070 con 0x7fe6700c1080 2024-03-25T02:00:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 10 monclient: dump: 2024-03-25T02:00:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.073 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67cdaf640 5 monclient: authenticate success, global_id 4312 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67cdaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 1 -- 172.21.15.83:0/1645066816 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6780a7df0 con 0x7fe6700c1080 2024-03-25T02:00:36.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67559a640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67cdaf640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.036+0000 7fe67cdaf640 4 set_mon_vals no callback set 2024-03-25T02:00:36.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f9987fff640 10 monclient(hunting): get_auth_request con 0x7f998813cef0 auth_method 0 2024-03-25T02:00:36.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f9987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f9987fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f9987fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fc7e6ffd640 1 --2- 172.21.15.83:0/2151176330 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e0038da0 0x7fc7e00c5250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc7d80098a0 tx=0x7fc7d802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff19d75176aac9dd server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67559a640 1 -- 172.21.15.83:0/1645066816 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6780af640 con 0x7fe6700c1080 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67559a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67559a640 10 monclient: dump: 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.076 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 10 monclient: shutdown 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 1 -- 172.21.15.83:0/1645066816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6700c1080 msgr2=0x7fe6700c1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 1 --2- 172.21.15.83:0/1645066816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6700c1080 0x7fe6700c1460 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe67806a5d0 tx=0x7fe6780a7190 comp rx=0 tx=0).stop 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 1 -- 172.21.15.83:0/1645066816 shutdown_connections 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 1 --2- 172.21.15.83:0/1645066816 >> v2:172.21.15.83:3300/0 conn(0x7fe6700069f0 0x7fe6700349a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 1 --2- 172.21.15.83:0/1645066816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6700c1080 0x7fe6700c1460 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7fe67cdaf640 1 -- 172.21.15.83:0/1645066816 >> 172.21.15.83:0/1645066816 conn(0x7fe670005ed0 msgr2=0x7fe6700062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 1 -- 172.21.15.83:0/3892219497 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c035020 con 0x7f2c20006a30 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 10 monclient: dump: 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.078 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.037+0000 7f2c25ffb640 1 -- 172.21.15.83:0/3892219497 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c1c031870 con 0x7f2c20006a30 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c25ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c25ffb640 1 -- 172.21.15.83:0/3892219497 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c031c40 con 0x7f2c20006a30 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c25ffb640 10 monclient: dump: 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.079 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c2da45640 5 monclient: authenticate success, global_id 4315 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c2da45640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c2da45640 4 set_mon_vals no callback set 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7fe67cdaf640 1 -- 172.21.15.83:0/1645066816 shutdown_connections 2024-03-25T02:00:36.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7fe67cdaf640 1 -- 172.21.15.83:0/1645066816 wait complete. 2024-03-25T02:00:36.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f8eaf7fe640 1 --2- 172.21.15.83:0/3114093314 >> v2:172.21.15.153:3300/0 conn(0x7f8eb0073230 0x7f8eb00737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f1ff77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ff00cc630 0x7f1ff00c0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f8eaf7fe640 10 monclient(hunting): get_auth_request con 0x7f8eb0073230 auth_method 0 2024-03-25T02:00:36.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f8eaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f8eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.081 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f8eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c2da45640 10 monclient: shutdown 2024-03-25T02:00:36.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c2da45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c2da45640 1 -- 172.21.15.83:0/3892219497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c20006a30 msgr2=0x7f2c200349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f2c2da45640 1 --2- 172.21.15.83:0/3892219497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c20006a30 0x7f2c200349e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c009870 tx=0x7f2c1c02b910 comp rx=0 tx=0).stop 2024-03-25T02:00:36.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f1ff77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ff00cc630 0x7f1ff00c0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40498/0 (socket says 172.21.15.83:40498) 2024-03-25T02:00:36.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f1ff77fe640 1 -- 172.21.15.83:0/375716917 learned_addr learned my addr 172.21.15.83:0/375716917 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f1ff77fe640 10 monclient(hunting): get_auth_request con 0x7f1ff00cc630 auth_method 0 2024-03-25T02:00:36.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f1ff77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f9987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f9987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f9987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f1ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.038+0000 7f1ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-03-25T02:00:36.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 1 -- 172.21.15.83:0/849601417 >> v1:172.21.15.153:6789/0 conn(0x7f65b00cf5c0 legacy=0x7f65b00cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 1 -- 172.21.15.83:0/849601417 >> v1:172.21.15.83:6789/0 conn(0x7f65b0034c20 legacy=0x7f65b00c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f65bd796640 1 -- 172.21.15.83:0/849601417 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b0033260 con 0x7f65b002fc10 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 10 monclient: monmap: 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.085 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 1 librados: starting msgr at 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 1 librados: starting objecter 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f2c2da45640 1 -- 172.21.15.83:0/3892219497 shutdown_connections 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f2c2da45640 1 --2- 172.21.15.83:0/3892219497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c20006a30 0x7f2c200349e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f2c2da45640 1 --2- 172.21.15.83:0/3892219497 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c200c10c0 0x7f2c200c14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.086 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f2c2da45640 1 -- 172.21.15.83:0/3892219497 >> 172.21.15.83:0/3892219497 conn(0x7f2c20005f10 msgr2=0x7f2c20006320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 1 Processor -- start 2024-03-25T02:00:36.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f2c2da45640 1 -- 172.21.15.83:0/3892219497 shutdown_connections 2024-03-25T02:00:36.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f2c2da45640 1 -- 172.21.15.83:0/3892219497 wait complete. 2024-03-25T02:00:36.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.087 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f1ff77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f1ff77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f1ff77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 1 -- start start 2024-03-25T02:00:36.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 1 librados: setting wanted keys 2024-03-25T02:00:36.088 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 1 librados: calling monclient init 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.039+0000 7fe67cdaf640 10 monclient: init 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f2c2da45640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f2c2da45640 10 monclient: monmap: 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.089 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6700069f0 0x7fe6700d14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe6700069f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe670031fb0 0x7fe670161a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe670031fb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f2c2da45640 1 librados: starting msgr at 2024-03-25T02:00:36.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f2c2da45640 1 librados: starting objecter 2024-03-25T02:00:36.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe6700c1080 0x7fe6700d4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe6700c1080 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe670032730 con 0x7fe6700069f0 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe6700c4350 con 0x7fe670031fb0 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fe670161290 con 0x7fe6700c1080 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7fe67cdaf640 10 monclient(hunting): authenticate will time out at 1355.230738s 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.040+0000 7f2c2da45640 1 Processor -- start 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 -- start start 2024-03-25T02:00:36.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 librados: setting wanted keys 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 librados: calling monclient init 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient: init 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 1 -- 172.21.15.83:0/1008587590 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f99880737f0 msgr2=0x7f99881430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 1 --2- 172.21.15.83:0/1008587590 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f99880737f0 0x7f99881430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T02:00:36.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c20006a30 0x7f2c200c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): picked mon.noname-a con 0x7f2c20006a30 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c20031ff0 0x7f2c2002b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): picked mon.noname-c con 0x7f2c20031ff0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c200c10c0 0x7f2c200b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): picked mon.noname-b con 0x7f2c200c10c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2c20032770 con 0x7f2c20006a30 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2c200c4390 con 0x7f2c200c10c0 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2c2002d520 con 0x7f2c20031ff0 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.094 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f2c2da45640 10 monclient(hunting): authenticate will time out at 1355.231637s 2024-03-25T02:00:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 1 -- 172.21.15.83:0/1008587590 >> v1:172.21.15.83:6789/0 conn(0x7f9988067500 legacy=0x7f9988071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 1 monclient: found mon.noname-a 2024-03-25T02:00:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.041+0000 7f9987fff640 1 -- 172.21.15.83:0/1008587590 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99881435e0 con 0x7f998813cef0 2024-03-25T02:00:36.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f1ff67fc640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0039010 0x7f1ff00c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 1 -- 172.21.15.83:0/3114093314 >> v1:172.21.15.153:6789/0 conn(0x7f8eb00aeca0 legacy=0x7f8eb01424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 1 -- 172.21.15.83:0/3114093314 >> v1:172.21.15.83:6789/0 conn(0x7f8eb0072e50 legacy=0x7f8eb013c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f8eaf7fe640 1 -- 172.21.15.83:0/3114093314 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb00af190 con 0x7f8eb0073230 2024-03-25T02:00:36.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.042+0000 7f65bd796640 1 --2- 172.21.15.83:0/849601417 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b002fc10 0x7f65b00c9680 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f65a80098a0 tx=0x7f65a802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c619ca0f75ddb2ac server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f974bfff640 1 -- 172.21.15.83:0/4199886810 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744035020 con 0x7f9754070630 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f974bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f974bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f974bfff640 10 monclient: dump: 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.097 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f974bfff640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f974bfff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f974bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f975b4e4640 5 monclient: authenticate success, global_id 4241 2024-03-25T02:00:36.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.046+0000 7f975b4e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 1 -- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff0035670 msgr2=0x7f1ff00c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f1ff0035670 0x7f1ff00c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 1 -- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0039010 msgr2=0x7f1ff00c22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1ff0039010 0x7f1ff00c22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.047+0000 7f1ff77fe640 1 -- 172.21.15.83:0/375716917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff0030360 con 0x7f1ff00cc630 2024-03-25T02:00:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.049+0000 7f9987fff640 1 --2- 172.21.15.83:0/1008587590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f998813cef0 0x7f998813d320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9980009d90 tx=0x7f9980030230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563224b613491d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.051+0000 7f1ff77fe640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ff00cc630 0x7f1ff00c0e90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ff806ad00 tx=0x7f1ff8068a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.051+0000 7f1fdffff640 10 client.?.objecter ms_handle_connect 0x7f1ff00cc630 2024-03-25T02:00:36.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.051+0000 7f1fdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.053+0000 7f974bfff640 1 -- 172.21.15.83:0/4199886810 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9744031800 con 0x7f9754070630 2024-03-25T02:00:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.053+0000 7f974bfff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.053+0000 7f975b4e4640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.053+0000 7f975b4e4640 4 set_mon_vals no callback set 2024-03-25T02:00:36.101 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7fe676d9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe6700c1080 0x7fe6700d4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f974bfff640 1 -- 172.21.15.83:0/4199886810 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744031bd0 con 0x7f9754070630 2024-03-25T02:00:36.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f974bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f974bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f974bfff640 10 monclient: dump: 2024-03-25T02:00:36.102 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.102 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.102 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 10 monclient: shutdown 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 1 -- 172.21.15.83:0/4199886810 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9754070630 msgr2=0x7f97540720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 1 --2- 172.21.15.83:0/4199886810 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9754070630 0x7f97540720c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97440097d0 tx=0x7f974402b8a0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 1 -- 172.21.15.83:0/4199886810 shutdown_connections 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 1 --2- 172.21.15.83:0/4199886810 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9754070630 0x7f97540720c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 1 -- 172.21.15.83:0/4199886810 >> 172.21.15.83:0/4199886810 conn(0x7f975406d0f0 msgr2=0x7f97540b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 1 -- 172.21.15.83:0/4199886810 shutdown_connections 2024-03-25T02:00:36.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.054+0000 7f975b4e4640 1 -- 172.21.15.83:0/4199886810 wait complete. 2024-03-25T02:00:36.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fe675d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe670031fb0 0x7fe670161a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f2c267fc640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c20031ff0 0x7f2c2002b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 1 -- 172.21.15.83:0/849601417 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8034070 con 0x7f65b002fc10 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: dump: 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.105 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 1 -- 172.21.15.83:0/849601417 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a8030840 con 0x7f65b002fc10 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 1 -- 172.21.15.83:0/849601417 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8030c10 con 0x7f65b002fc10 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bcf95640 10 monclient: dump: 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.106 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bef99640 5 monclient: authenticate success, global_id 4247 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bef99640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f65bef99640 4 set_mon_vals no callback set 2024-03-25T02:00:36.107 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 1 -- 172.21.15.83:0/2151176330 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80317c0 con 0x7fc7e0038da0 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 10 monclient: dump: 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.108 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e7fff640 5 monclient: authenticate success, global_id 4244 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 1 -- 172.21.15.83:0/2151176330 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d8037070 con 0x7fc7e0038da0 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e5ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e7fff640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7fc7e7fff640 4 set_mon_vals no callback set 2024-03-25T02:00:36.109 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f975b4e4640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 10 monclient: shutdown 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 1 -- 172.21.15.83:0/849601417 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b002fc10 msgr2=0x7f65b00c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 1 --2- 172.21.15.83:0/849601417 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b002fc10 0x7f65b00c9680 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f65a80098a0 tx=0x7f65a802b980 comp rx=0 tx=0).stop 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fe676d9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe6700c1080 0x7fe6700d4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54444/0 (socket says 172.21.15.83:54444) 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fe676d9d640 1 -- 172.21.15.83:0/2725632370 learned_addr learned my addr 172.21.15.83:0/2725632370 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fe676d9d640 10 monclient(hunting): get_auth_request con 0x7fe6700c1080 auth_method 0 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fe676d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fe676d9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fe676d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f975b4e4640 10 monclient: monmap: 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.111 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f975b4e4640 1 librados: starting msgr at 2024-03-25T02:00:36.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.055+0000 7f975b4e4640 1 librados: starting objecter 2024-03-25T02:00:36.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 Processor -- start 2024-03-25T02:00:36.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 1 -- 172.21.15.83:0/849601417 shutdown_connections 2024-03-25T02:00:36.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 1 --2- 172.21.15.83:0/849601417 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b002fc10 0x7f65b00c9680 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 1 -- 172.21.15.83:0/849601417 >> 172.21.15.83:0/849601417 conn(0x7f65b00060c0 msgr2=0x7f65b00064d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.112 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 -- start start 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 librados: setting wanted keys 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 librados: calling monclient init 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient: init 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e5ffb640 1 -- 172.21.15.83:0/2151176330 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80317c0 con 0x7fc7e0038da0 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e5ffb640 10 monclient: dump: 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.113 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f2c267fc640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c20031ff0 0x7f2c2002b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54452/0 (socket says 172.21.15.83:54452) 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f2c267fc640 1 -- 172.21.15.83:0/85110343 learned_addr learned my addr 172.21.15.83:0/85110343 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f2c267fc640 10 monclient(hunting): get_auth_request con 0x7f2c20031ff0 auth_method 0 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f2c267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.114 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f2c267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f2c267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 1 -- 172.21.15.83:0/849601417 shutdown_connections 2024-03-25T02:00:36.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f65bef99640 1 -- 172.21.15.83:0/849601417 wait complete. 2024-03-25T02:00:36.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 10 monclient: shutdown 2024-03-25T02:00:36.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 -- 172.21.15.83:0/2151176330 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e0038da0 msgr2=0x7fc7e00c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.115 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 --2- 172.21.15.83:0/2151176330 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e0038da0 0x7fc7e00c5250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc7d80098a0 tx=0x7fc7d802b860 comp rx=0 tx=0).stop 2024-03-25T02:00:36.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:00:36.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f975406b870 0x7f97540aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): picked mon.noname-b con 0x7f975406b870 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.116 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 -- 172.21.15.83:0/2151176330 shutdown_connections 2024-03-25T02:00:36.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f97541446b0 0x7f975414aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): picked mon.noname-c con 0x7f97541446b0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 --2- 172.21.15.83:0/2151176330 >> v2:172.21.15.83:3300/0 conn(0x7fc7e0039180 0x7fc7e00c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 --2- 172.21.15.83:0/2151176330 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e0038da0 0x7fc7e00c5250 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 -- 172.21.15.83:0/2151176330 >> 172.21.15.83:0/2151176330 conn(0x7fc7e0006600 msgr2=0x7fc7e0006a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.117 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97540a94e0 0x7f97540abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): picked mon.noname-a con 0x7f97540a94e0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9754144ce0 con 0x7f97540a94e0 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9754144b60 con 0x7f975406b870 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f9754144e60 con 0x7f97541446b0 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7f975b4e4640 10 monclient(hunting): authenticate will time out at 1355.246766s 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 -- 172.21.15.83:0/2151176330 shutdown_connections 2024-03-25T02:00:36.118 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.056+0000 7fc7e7fff640 1 -- 172.21.15.83:0/2151176330 wait complete. 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 10 monclient: monmap: 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.119 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f2c26ffd640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c20006a30 0x7f2c200c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.120 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8099070 con 0x7f1ff00cc630 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: dump: 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.121 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff806dc60 con 0x7f1ff00cc630 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 5 monclient: authenticate success, global_id 4324 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8094610 con 0x7f1ff00cc630 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1fdffff640 10 monclient: dump: 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.122 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 10 monclient: _renew_subs 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ff0009ec0 con 0x7f1ff00cc630 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ff57fa640 4 set_mon_vals no callback set 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 10 monclient: _renew_subs 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ff00c1d90 con 0x7f1ff00cc630 2024-03-25T02:00:36.123 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 1 librados: starting msgr at 2024-03-25T02:00:36.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 1 librados: starting objecter 2024-03-25T02:00:36.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 1 Processor -- start 2024-03-25T02:00:36.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f1ffd825640 1 librados: init done 2024-03-25T02:00:36.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fe675d9b640 10 monclient(hunting): get_auth_request con 0x7fe670031fb0 auth_method 0 2024-03-25T02:00:36.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fe675d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fe675d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fe675d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 1 -- start start 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 1 librados: setting wanted keys 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 1 librados: calling monclient init 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f65bef99640 10 monclient: init 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fc7e7fff640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fc7e7fff640 10 monclient: monmap: 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.125 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f2c26ffd640 10 monclient(hunting): get_auth_request con 0x7f2c20006a30 auth_method 0 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f2c26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fc7e7fff640 1 librados: starting msgr at 2024-03-25T02:00:36.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7fc7e7fff640 1 librados: starting objecter 2024-03-25T02:00:36.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f2c26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.057+0000 7f2c26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f2c267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f2c267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7fc7e7fff640 1 Processor -- start 2024-03-25T02:00:36.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f2c267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7fc7e7fff640 1 -- start start 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65b002fc10 0x7f65b00c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): picked mon.noname-b con 0x7f65b002fc10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65b0034c20 0x7f65b00c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): picked mon.noname-a con 0x7f65b0034c20 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b00cf5c0 0x7f65b00cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): picked mon.noname-c con 0x7f65b00cf5c0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65b00d7ae0 con 0x7f65b0034c20 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65b00d7960 con 0x7f65b002fc10 2024-03-25T02:00:36.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f65b00d7c60 con 0x7f65b00cf5c0 2024-03-25T02:00:36.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f65bef99640 10 monclient(hunting): authenticate will time out at 1355.248414s 2024-03-25T02:00:36.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7fc7e7fff640 1 librados: setting wanted keys 2024-03-25T02:00:36.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7fc7e7fff640 1 librados: calling monclient init 2024-03-25T02:00:36.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7fc7e7fff640 10 monclient: init 2024-03-25T02:00:36.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f1ffd825640 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f1ffd825640 15 client.4324.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f1ffd825640 20 client.4324.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f1ffd825640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f1ffd825640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1ff0008620 con 0x7f1ff00cc630 2024-03-25T02:00:36.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7f8eaf7fe640 1 --2- 172.21.15.83:0/3114093314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb0073230 0x7f8eb00737d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097a0 tx=0x7f8ea402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6de9c16de38eecc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.058+0000 7fc7e7fff640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7e0038da0 0x7fc7e0164b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc7e0038da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e00c5790 0x7fc7e0009d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc7e00c5790 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7e000a2d0 0x7fc7e00c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc7e000a2d0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e00c8880 con 0x7fc7e000a2d0 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e00366c0 con 0x7fc7e0038da0 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fc7e00c8a00 con 0x7fc7e00c5790 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.059+0000 7fc7e7fff640 10 monclient(hunting): authenticate will time out at 1355.249047s 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.060+0000 7fe67659c640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6700069f0 0x7fe6700d14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.133 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.061+0000 7fe67659c640 10 monclient(hunting): get_auth_request con 0x7fe6700069f0 auth_method 0 2024-03-25T02:00:36.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.061+0000 7fe67659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.061+0000 7fe67659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.061+0000 7fe67659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.061+0000 7f9758a7f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f97541446b0 0x7f975414aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7fe676d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7fe676d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.134 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7fe676d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7fe67659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7fe67659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7fe67659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f9758a7f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f97541446b0 0x7f975414aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54458/0 (socket says 172.21.15.83:54458) 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f9758a7f640 1 -- 172.21.15.83:0/2545748317 learned_addr learned my addr 172.21.15.83:0/2545748317 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f9758a7f640 10 monclient(hunting): get_auth_request con 0x7f97541446b0 auth_method 0 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f9758a7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f9758a7f640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.135 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f9758a7f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f2c26ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f2c26ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.062+0000 7f2c26ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 1 -- 172.21.15.83:0/85110343 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c20031ff0 msgr2=0x7f2c2002b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2c20031ff0 0x7f2c2002b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 1 -- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c200c10c0 msgr2=0x7f2c200b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2c200c10c0 0x7f2c200b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 1 monclient: found mon.noname-a 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7f2c26ffd640 1 -- 172.21.15.83:0/85110343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2002d800 con 0x7f2c20006a30 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 1 -- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe670031fb0 msgr2=0x7fe670161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe670031fb0 0x7fe670161a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 1 -- 172.21.15.83:0/2725632370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6700069f0 msgr2=0x7fe6700d14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6700069f0 0x7fe6700d14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.065+0000 7fe676d9d640 1 -- 172.21.15.83:0/2725632370 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe670161f40 con 0x7fe6700c1080 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.066+0000 7f9758a7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.066+0000 7f9758a7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.066+0000 7f9758a7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.067+0000 7f9759a81640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97540a94e0 0x7f97540abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 1 -- 172.21.15.83:0/1008587590 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998003d070 con 0x7f998813cef0 2024-03-25T02:00:36.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9759a81640 10 monclient(hunting): get_auth_request con 0x7f97540a94e0 auth_method 0 2024-03-25T02:00:36.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9759a81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9759a81640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9759a81640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: dump: 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.140 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 1 -- 172.21.15.83:0/1008587590 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9980038440 con 0x7f998813cef0 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 1 -- 172.21.15.83:0/1008587590 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99800387f0 con 0x7f998813cef0 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9985ffb640 10 monclient: dump: 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.141 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f998d359640 5 monclient: authenticate success, global_id 4299 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f998d359640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f998d359640 4 set_mon_vals no callback set 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9759a81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9759a81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f9759a81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f998d359640 10 monclient: shutdown 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f998d359640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f998d359640 1 -- 172.21.15.83:0/1008587590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f998813cef0 msgr2=0x7f998813d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.068+0000 7f998d359640 1 --2- 172.21.15.83:0/1008587590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f998813cef0 0x7f998813d320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9980009d90 tx=0x7f9980030230 comp rx=0 tx=0).stop 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 1 -- 172.21.15.83:0/1008587590 shutdown_connections 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 1 --2- 172.21.15.83:0/1008587590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f998813cef0 0x7f998813d320 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 1 --2- 172.21.15.83:0/1008587590 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f99880737f0 0x7f99881430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 1 -- 172.21.15.83:0/1008587590 >> 172.21.15.83:0/1008587590 conn(0x7f998806b7e0 msgr2=0x7f998806bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 1 -- 172.21.15.83:0/1008587590 shutdown_connections 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 1 -- 172.21.15.83:0/1008587590 wait complete. 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.069+0000 7f998d359640 10 monclient: monmap: 2024-03-25T02:00:36.144 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 1 librados: starting msgr at 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 1 librados: starting objecter 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 1 Processor -- start 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 1 -- start start 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 1 librados: setting wanted keys 2024-03-25T02:00:36.145 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 1 librados: calling monclient init 2024-03-25T02:00:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 10 monclient: init 2024-03-25T02:00:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f2c26ffd640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c20006a30 0x7f2c200c7370 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c28050bb0 tx=0x7f2c28068a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f2c0ffff640 10 client.?.objecter ms_handle_connect 0x7f2c20006a30 2024-03-25T02:00:36.146 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f2c0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9988067500 0x7f99880adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.070+0000 7f998d359640 10 monclient(hunting): picked mon.noname-b con 0x7f9988067500 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99880737f0 0x7f99881d8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 10 monclient(hunting): picked mon.noname-a con 0x7f99880737f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f998813cef0 0x7f99881d7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 10 monclient(hunting): picked mon.noname-c con 0x7f998813cef0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99880712a0 con 0x7f99880737f0 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9988071660 con 0x7f9988067500 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f9988071420 con 0x7f998813cef0 2024-03-25T02:00:36.148 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f998d359640 10 monclient(hunting): authenticate will time out at 1355.261169s 2024-03-25T02:00:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-03-25T02:00:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 1 -- 172.21.15.83:0/2545748317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f975406b870 msgr2=0x7f97540aaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f975406b870 0x7f97540aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.149 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 1 -- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97540a94e0 msgr2=0x7f97540abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97540a94e0 0x7f97540abc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.071+0000 7f9758a7f640 1 -- 172.21.15.83:0/2545748317 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97540ab4c0 con 0x7f97541446b0 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.073+0000 7fe676d9d640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe6700c1080 0x7fe6700d4cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c390 tx=0x7fe66c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.073+0000 7fe666ffd640 10 client.?.objecter ms_handle_connect 0x7fe6700c1080 2024-03-25T02:00:36.150 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.073+0000 7fe666ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.080+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f1ff80947b0 con 0x7f1ff00cc630 2024-03-25T02:00:36.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.080+0000 7f1fdffff640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f1fe406da20 0x7f1fe406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.080+0000 7f1ff6ffd640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f1fe406da20 0x7f1fe406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.080+0000 7f1ff6ffd640 10 monclient: get_auth_request con 0x7f1fe406da20 auth_method 0 2024-03-25T02:00:36.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.081+0000 7fc7e77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7e000a2d0 0x7fc7e00c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.081+0000 7f1fdffff640 10 client.4324.objecter ms_handle_connect 0x7f1fe406da20 2024-03-25T02:00:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.081+0000 7f1ff6ffd640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f1fe406da20 0x7f1fe406fee0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1fec001560 tx=0x7f1fec007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.083+0000 7fc7e67fc640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e00c5790 0x7fc7e0009d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.083+0000 7f9987fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f998813cef0 0x7f99881d7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7fc7e67fc640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e00c5790 0x7fc7e0009d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54474/0 (socket says 172.21.15.83:54474) 2024-03-25T02:00:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7fc7e67fc640 1 -- 172.21.15.83:0/3422872661 learned_addr learned my addr 172.21.15.83:0/3422872661 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7fc7e67fc640 10 monclient(hunting): get_auth_request con 0x7fc7e00c5790 auth_method 0 2024-03-25T02:00:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7fc7e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7fc7e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7fc7e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f1ff80d2570 con 0x7f1ff00cc630 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 10 client.4324.objecter ms_dispatch 0x7f1ff8000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 3 client.4324.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 3 client.4324.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 10 client.4324.objecter checking command 1 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7f1fe4073e60 0x7f1fe4076280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 20 client.4324.objecter _get_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 20 client.4324.objecter put_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 15 client.4324.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 20 client.4324.objecter _get_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 20 client.4324.objecter get_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 15 client.4324.objecter _session_command_op_assign 2 1 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 20 client.4324.objecter put_session s=0x7f1fe4073790 osd=2 4 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 10 client.4324.objecter _send_command 1 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 --> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fe4076930 con 0x7f1fe4073e60 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1ff80d0770 con 0x7f1ff00cc630 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f1fdffff640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f9987fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f998813cef0 0x7f99881d7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54480/0 (socket says 172.21.15.83:54480) 2024-03-25T02:00:36.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f9987fff640 1 -- 172.21.15.83:0/3177696750 learned_addr learned my addr 172.21.15.83:0/3177696750 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f9987fff640 10 monclient(hunting): get_auth_request con 0x7f998813cef0 auth_method 0 2024-03-25T02:00:36.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f9987fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f9987fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.084+0000 7f9987fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.085+0000 7f1ff67fc640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7f1fe4073e60 0x7f1fe4076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.085+0000 7f9987fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.085+0000 7f9987fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.085+0000 7f9987fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.085+0000 7f1ff67fc640 10 monclient: get_auth_request con 0x7f1fe4073e60 auth_method 0 2024-03-25T02:00:36.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.085+0000 7f65bd796640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65b0034c20 0x7f65b00c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.085+0000 7fc7e77fe640 10 monclient(hunting): get_auth_request con 0x7fc7e000a2d0 auth_method 0 2024-03-25T02:00:36.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7fc7e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7fc7e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7fc7e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 1 -- 172.21.15.83:0/3114093314 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4035020 con 0x7f8eb0073230 2024-03-25T02:00:36.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: dump: 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eb5792640 5 monclient: authenticate success, global_id 4321 2024-03-25T02:00:36.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eb5792640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 1 -- 172.21.15.83:0/3114093314 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea4031830 con 0x7f8eb0073230 2024-03-25T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eb5792640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eb5792640 4 set_mon_vals no callback set 2024-03-25T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f65bd796640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65b0034c20 0x7f65b00c9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55958/0 (socket says 172.21.15.83:55958) 2024-03-25T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f65bd796640 1 -- 172.21.15.83:0/2352701376 learned_addr learned my addr 172.21.15.83:0/2352701376 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 1 -- 172.21.15.83:0/3114093314 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4035020 con 0x7f8eb0073230 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f8eaeffd640 10 monclient: dump: 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.161 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f65bd796640 10 monclient(hunting): get_auth_request con 0x7f65b0034c20 auth_method 0 2024-03-25T02:00:36.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f65bd796640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f65bd796640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f65bd796640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f9758a7f640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f97541446b0 0x7f975414aa70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f974402beb0 tx=0x7f97440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f9749ffb640 10 client.?.objecter ms_handle_connect 0x7f97541446b0 2024-03-25T02:00:36.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.086+0000 7f9749ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 10 monclient: shutdown 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 1 -- 172.21.15.83:0/3114093314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb0073230 msgr2=0x7f8eb00737d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 1 --2- 172.21.15.83:0/3114093314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb0073230 0x7f8eb00737d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097a0 tx=0x7f8ea402b8d0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 1 -- 172.21.15.83:0/3114093314 shutdown_connections 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 1 --2- 172.21.15.83:0/3114093314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb0073230 0x7f8eb00737d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 1 -- 172.21.15.83:0/3114093314 >> 172.21.15.83:0/3114093314 conn(0x7f8eb00b4b80 msgr2=0x7f8eb00b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 1 -- 172.21.15.83:0/3114093314 shutdown_connections 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 1 -- 172.21.15.83:0/3114093314 wait complete. 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.087+0000 7f8eb5792640 10 monclient: monmap: 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.164 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.088+0000 7f8eb5792640 1 librados: starting msgr at 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.088+0000 7f8eb5792640 1 librados: starting objecter 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.088+0000 7f8eb5792640 1 Processor -- start 2024-03-25T02:00:36.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.088+0000 7fc7e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.088+0000 7fc7e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.088+0000 7fc7e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f8eb5792640 1 -- start start 2024-03-25T02:00:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f8eb5792640 1 librados: setting wanted keys 2024-03-25T02:00:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f8eb5792640 1 librados: calling monclient init 2024-03-25T02:00:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f8eb5792640 10 monclient: init 2024-03-25T02:00:36.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7fc7e6ffd640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7e0038da0 0x7fc7e0164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f9986ffd640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9988067500 0x7f99880adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7fc7e6ffd640 10 monclient(hunting): get_auth_request con 0x7fc7e0038da0 auth_method 0 2024-03-25T02:00:36.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7fc7e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.167 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7fc7e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7fc7e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f8eb5792640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f8eb5792640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:36.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.089+0000 7f8eb5792640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f8eb0072e50 0x7f8eb0132db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb0072e50 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb014b0c0 0x7f8eb014b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f1ff67fc640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7f1fe4073e60 0x7f1fe4076280 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb014b0c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f1fdffff640 10 client.4324.objecter ms_handle_connect 0x7f1fe4073e60 2024-03-25T02:00:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f9986ffd640 10 monclient(hunting): get_auth_request con 0x7f9988067500 auth_method 0 2024-03-25T02:00:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f9986ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f9986ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f9986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8eb012d8d0 0x7f8eb012dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb012d8d0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb00af420 con 0x7f8eb012d8d0 2024-03-25T02:00:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb0071850 con 0x7f8eb014b0c0 2024-03-25T02:00:36.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb014a870 con 0x7f8eb0072e50 2024-03-25T02:00:36.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f8eb5792640 10 monclient(hunting): authenticate will time out at 1355.280399s 2024-03-25T02:00:36.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28094420 con 0x7f2c20006a30 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: dump: 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c2808f030 con 0x7f2c20006a30 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28094800 con 0x7f2c20006a30 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c0ffff640 10 monclient: dump: 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.174 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c257fa640 4 set_mon_vals no callback set 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 5 monclient: authenticate success, global_id 4311 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 10 monclient: _renew_subs 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c2002e240 con 0x7f2c20006a30 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 10 monclient: _renew_subs 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2002b720 con 0x7f2c20006a30 2024-03-25T02:00:36.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.090+0000 7f2c2da45640 1 librados: init done 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7f2c2da45640 10 client.4311.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7f2c2da45640 15 client.4311.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7f2c2da45640 20 client.4311.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7f2c2da45640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7f2c2da45640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.091+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2c2002ea60 con 0x7f2c20006a30 2024-03-25T02:00:36.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.093+0000 7fc7e6ffd640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-03-25T02:00:36.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.093+0000 7fc7e6ffd640 1 -- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e00c5790 msgr2=0x7fc7e0009d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.093+0000 7fc7e6ffd640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fc7e00c5790 0x7fc7e0009d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7fc7e6ffd640 1 -- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7e000a2d0 msgr2=0x7fc7e00c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7fc7e6ffd640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7e000a2d0 0x7fc7e00c39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7fc7e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7fc7e6ffd640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7fc7e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7fc7e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7fc7e6ffd640 1 -- 172.21.15.83:0/3422872661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e000a710 con 0x7fc7e0038da0 2024-03-25T02:00:36.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 1 -- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9988067500 msgr2=0x7f99880adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9988067500 0x7f99880adaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 1 -- 172.21.15.83:0/3177696750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99880737f0 msgr2=0x7f99881d8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99880737f0 0x7f99881d8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9987fff640 1 -- 172.21.15.83:0/3177696750 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99880ae590 con 0x7f998813cef0 2024-03-25T02:00:36.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.094+0000 7f9986ffd640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9988067500 0x7f99880adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-25T02:00:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.095+0000 7f65be798640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b00cf5c0 0x7f65b00cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bdf97640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65b002fc10 0x7f65b00c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bd796640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bd796640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bd796640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7fc7cf7fe640 10 client.?.objecter ms_handle_connect 0x7fc7e0038da0 2024-03-25T02:00:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7fc7cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7fc7e6ffd640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7e0038da0 0x7fc7e0164b40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8033000 tx=0x7fc7d80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bdf97640 10 monclient(hunting): get_auth_request con 0x7f65b002fc10 auth_method 0 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bdf97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bdf97640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.096+0000 7f65bdf97640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.097+0000 7f65bdf97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.097+0000 7f65bdf97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.097+0000 7f65bdf97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.099+0000 7f65be798640 10 monclient(hunting): get_auth_request con 0x7f65b00cf5c0 auth_method 0 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.099+0000 7f65be798640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.099+0000 7f65be798640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.099+0000 7f65be798640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.099+0000 7f9987fff640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f998813cef0 0x7f99881d7890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f998003b040 tx=0x7f9980004a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.099+0000 7f996f7fe640 10 client.?.objecter ms_handle_connect 0x7f998813cef0 2024-03-25T02:00:36.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.099+0000 7f996f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.100+0000 7f65be798640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65be798640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65be798640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 1 -- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b00cf5c0 msgr2=0x7f65b00cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f65b00cf5c0 0x7f65b00cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 1 -- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65b0034c20 msgr2=0x7f65b00c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65b0034c20 0x7f65b00c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.101+0000 7f65bdf97640 1 -- 172.21.15.83:0/2352701376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b00cd8c0 con 0x7f65b002fc10 2024-03-25T02:00:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744041070 con 0x7f97541446b0 2024-03-25T02:00:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: dump: 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974403f070 con 0x7f97541446b0 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744035040 con 0x7f97541446b0 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f9749ffb640 10 monclient: dump: 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 5 monclient: authenticate success, global_id 4256 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 10 monclient: _renew_subs 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97540ab750 con 0x7f97541446b0 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f974b7fe640 4 set_mon_vals no callback set 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 10 monclient: _renew_subs 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f975414afb0 con 0x7f97541446b0 2024-03-25T02:00:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7f975b4e4640 1 librados: init done 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c017070 con 0x7fe6700c1080 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 10 monclient: dump: 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe66c0035f0 con 0x7fe6700c1080 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 5 monclient: authenticate success, global_id 4253 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 10 monclient: _renew_subs 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6700d4560 con 0x7fe6700c1080 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 10 monclient: _renew_subs 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6700d53d0 con 0x7fe6700c1080 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe67cdaf640 1 librados: init done 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe666ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.103+0000 7fe674d99640 4 set_mon_vals no callback set 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0039e0 con 0x7fe6700c1080 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe666ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe666ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe666ffd640 10 monclient: dump: 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7f975b4e4640 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7f975b4e4640 15 client.4256.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7f975b4e4640 20 client.4256.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7f975b4e4640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7f975b4e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f97540ad420 con 0x7f97541446b0 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe67cdaf640 10 client.4253.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe67cdaf640 15 client.4253.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe67cdaf640 20 client.4253.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe67cdaf640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe67cdaf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.104+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe65c002910 con 0x7fe6700c1080 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.105+0000 7f65bdf97640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65b002fc10 0x7f65b00c4940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65b400c340 tx=0x7f65b400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.105+0000 7f65ae7fc640 10 client.?.objecter ms_handle_connect 0x7f65b002fc10 2024-03-25T02:00:36.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.105+0000 7f65ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8041070 con 0x7fc7e0038da0 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: dump: 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.195 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d8003e50 con 0x7fc7e0038da0 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80317c0 con 0x7fc7e0038da0 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7cf7fe640 10 monclient: dump: 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 5 monclient: authenticate success, global_id 4330 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 10 monclient: _renew_subs 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e000bd40 con 0x7fc7e0038da0 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 10 monclient: _renew_subs 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e0165660 con 0x7fc7e0038da0 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e7fff640 1 librados: init done 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.106+0000 7fc7e4ff9640 4 set_mon_vals no callback set 2024-03-25T02:00:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.107+0000 7fc7e7fff640 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.107+0000 7fc7e7fff640 15 client.4330.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.107+0000 7fc7e7fff640 20 client.4330.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.107+0000 7fc7e7fff640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.107+0000 7fc7e7fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.107+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc7e0008620 con 0x7fc7e0038da0 2024-03-25T02:00:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.110+0000 7f8eaf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb014b0c0 0x7f8eb014b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.115+0000 7f8eb4f91640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8eb012d8d0 0x7f8eb012dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.115+0000 7f8eb4f91640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8eb012d8d0 0x7f8eb012dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:55990/0 (socket says 172.21.15.83:55990) 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.115+0000 7f8eb4f91640 1 -- 172.21.15.83:0/3299155400 learned_addr learned my addr 172.21.15.83:0/3299155400 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.115+0000 7f8eb4f91640 10 monclient(hunting): get_auth_request con 0x7f8eb012d8d0 auth_method 0 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.115+0000 7f8eb4f91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.115+0000 7f8eb4f91640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.115+0000 7f8eb4f91640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.116+0000 7f8eb4f91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.116+0000 7f8eb4f91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.116+0000 7f8eb4f91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.117+0000 7f8eaf7fe640 10 monclient(hunting): get_auth_request con 0x7f8eb014b0c0 auth_method 0 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.117+0000 7f8eaf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.117+0000 7f8eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.117+0000 7f8eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.130+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f2c280933d0 con 0x7f2c20006a30 2024-03-25T02:00:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.130+0000 7f2c0ffff640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2c1406da50 0x7f2c1406ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.130+0000 7f2c267fc640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2c1406da50 0x7f2c1406ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f2c267fc640 10 monclient: get_auth_request con 0x7f2c1406da50 auth_method 0 2024-03-25T02:00:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f8eaffff640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f8eb0072e50 0x7f8eb0132db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f2c267fc640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2c1406da50 0x7f2c1406ff10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0090b0 tx=0x7f2c1c03c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f2c0ffff640 10 client.4311.objecter ms_handle_connect 0x7f2c1406da50 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f998003d050 con 0x7f998813cef0 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: dump: 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9980049070 con 0x7f998813cef0 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9980036d40 con 0x7f998813cef0 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fc7d804c060 con 0x7fc7e0038da0 2024-03-25T02:00:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f996f7fe640 10 monclient: dump: 2024-03-25T02:00:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 5 monclient: authenticate success, global_id 4259 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 10 monclient: _renew_subs 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f99857fa640 4 set_mon_vals no callback set 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 1 -- 172.21.15.83:0/3177696750 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9988140890 con 0x7f998813cef0 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 10 monclient: _renew_subs 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 1 -- 172.21.15.83:0/3177696750 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f99880a96c0 con 0x7f998813cef0 2024-03-25T02:00:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.131+0000 7f998d359640 1 librados: init done 2024-03-25T02:00:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7fc7cf7fe640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fc7d406da50 0x7fc7d406ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7fc7e67fc640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fc7d406da50 0x7fc7d406ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7fc7e67fc640 10 monclient: get_auth_request con 0x7fc7d406da50 auth_method 0 2024-03-25T02:00:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7fc7cf7fe640 10 client.4330.objecter ms_handle_connect 0x7fc7d406da50 2024-03-25T02:00:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7fc7e67fc640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fc7d406da50 0x7fc7d406ff10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000e170 tx=0x7fc7dc008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7f998d359640 10 client.4259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7f998d359640 15 client.4259.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7f998d359640 20 client.4259.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7f998d359640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7f998d359640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.132+0000 7f998d359640 1 -- 172.21.15.83:0/3177696750 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f99881d7e60 con 0x7f998813cef0 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fc7d80789b0 con 0x7fc7e0038da0 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 10 client.4330.objecter ms_dispatch 0x7fc7e8000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 3 client.4330.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 3 client.4330.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 10 client.4330.objecter checking command 1 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7fc7d4073e90 0x7fc7d40762b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 20 client.4330.objecter _get_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 20 client.4330.objecter put_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 15 client.4330.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 20 client.4330.objecter _get_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 20 client.4330.objecter get_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 15 client.4330.objecter _session_command_op_assign 0 1 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 20 client.4330.objecter put_session s=0x7fc7d40737c0 osd=0 4 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 10 client.4330.objecter _send_command 1 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 --> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7d4076960 con 0x7fc7d4073e90 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc7d8076bb0 con 0x7fc7e0038da0 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7fc7cf7fe640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.134+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f2c28068c70 con 0x7f2c20006a30 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 10 client.4311.objecter ms_dispatch 0x7f2c28000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 3 client.4311.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 3 client.4311.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b40052d0 con 0x7f65b002fc10 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: dump: 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.210 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 5 monclient: authenticate success, global_id 4336 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 10 client.4311.objecter checking command 1 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7f2c14073e90 0x7f2c140762b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 20 client.4311.objecter _get_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 20 client.4311.objecter put_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 15 client.4311.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.211 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 20 client.4311.objecter _get_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 20 client.4311.objecter get_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 15 client.4311.objecter _session_command_op_assign 6 1 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 20 client.4311.objecter put_session s=0x7f2c140737c0 osd=6 4 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 10 client.4311.objecter _send_command 1 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 --> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c14076960 con 0x7f2c14073e90 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2c280d0810 con 0x7f2c20006a30 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c0ffff640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 10 monclient: _renew_subs 2024-03-25T02:00:36.212 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b00cc390 con 0x7f65b002fc10 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 10 monclient: _renew_subs 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b00cc8a0 con 0x7f65b002fc10 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65bef99640 1 librados: init done 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7fc7e77fe640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7fc7d4073e90 0x7fc7d40762b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7fc7e77fe640 10 monclient: get_auth_request con 0x7fc7d4073e90 auth_method 0 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b4005490 con 0x7f65b002fc10 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b4007260 con 0x7f65b002fc10 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65ae7fc640 10 monclient: dump: 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.214 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.215 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.215 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.215 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f65affff640 4 set_mon_vals no callback set 2024-03-25T02:00:36.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.135+0000 7f2c277fe640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7f2c14073e90 0x7f2c140762b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.136+0000 7f2c277fe640 10 monclient: get_auth_request con 0x7f2c14073e90 auth_method 0 2024-03-25T02:00:36.215 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.136+0000 7f65bef99640 10 client.4336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.136+0000 7f65bef99640 15 client.4336.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.136+0000 7f65bef99640 20 client.4336.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.136+0000 7f65bef99640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.136+0000 7f65bef99640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.136+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f65a0002910 con 0x7f65b002fc10 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 1 -- 172.21.15.83:0/3299155400 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f8eb0072e50 msgr2=0x7f8eb0132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f8eb0072e50 0x7f8eb0132db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 1 -- 172.21.15.83:0/3299155400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8eb012d8d0 msgr2=0x7f8eb012dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8eb012d8d0 0x7f8eb012dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.137+0000 7f8eaf7fe640 1 -- 172.21.15.83:0/3299155400 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb00a97f0 con 0x7f8eb014b0c0 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.138+0000 7fc7cf7fe640 10 client.4330.objecter ms_handle_connect 0x7fc7d4073e90 2024-03-25T02:00:36.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.138+0000 7fc7e77fe640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7fc7d4073e90 0x7fc7d40762b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.140+0000 7f2c0ffff640 10 client.4311.objecter ms_handle_connect 0x7f2c14073e90 2024-03-25T02:00:36.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.140+0000 7f2c277fe640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7f2c14073e90 0x7f2c140762b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.142+0000 7f8eacff9640 10 client.?.objecter ms_handle_connect 0x7f8eb014b0c0 2024-03-25T02:00:36.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.142+0000 7f8eacff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.142+0000 7f8eaf7fe640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb014b0c0 0x7f8eb014b530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8eb006c650 tx=0x7f8ea40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.148+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== osd.2 v2:172.21.15.83:6816/602002618 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1fe4076930 con 0x7f1fe4073e60 2024-03-25T02:00:36.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.148+0000 7f1fdffff640 10 client.4324.objecter ms_dispatch 0x7f1ff8000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.148+0000 7f1fdffff640 10 client.4324.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.148+0000 7f1fdffff640 20 client.4324.objecter put_session s=0x7f1fe4073790 osd=2 4 2024-03-25T02:00:36.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.148+0000 7f1fdffff640 15 client.4324.objecter _session_command_op_remove 2 1 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.150+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f9744038020 con 0x7f97541446b0 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.150+0000 7f9749ffb640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f973806da70 0x7f973806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.151+0000 7f9759280640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f973806da70 0x7f973806ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.151+0000 7f9759280640 10 monclient: get_auth_request con 0x7f973806da70 auth_method 0 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.151+0000 7f9749ffb640 10 client.4256.objecter ms_handle_connect 0x7f973806da70 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.151+0000 7f9759280640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f973806da70 0x7f973806ff30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f97500015f0 tx=0x7f9750007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f9744079690 con 0x7f97541446b0 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 10 client.4256.objecter ms_dispatch 0x7f9754000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 3 client.4256.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 3 client.4256.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 10 client.4256.objecter checking command 1 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f9738073eb0 0x7f97380762d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 20 client.4256.objecter _get_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 20 client.4256.objecter put_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 15 client.4256.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 20 client.4256.objecter _get_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 20 client.4256.objecter get_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 15 client.4256.objecter _session_command_op_assign 1 1 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 20 client.4256.objecter put_session s=0x7f97380737e0 osd=1 4 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 10 client.4256.objecter _send_command 1 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 --> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9744033ec0 con 0x7f9738073eb0 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9744077890 con 0x7f97541446b0 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.153+0000 7f9749ffb640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.154+0000 7f9759a81640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f9738073eb0 0x7f97380762d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.155+0000 7f9759a81640 10 monclient: get_auth_request con 0x7f9738073eb0 auth_method 0 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.159+0000 7f9749ffb640 10 client.4256.objecter ms_handle_connect 0x7f9738073eb0 2024-03-25T02:00:36.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.159+0000 7f9759a81640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f9738073eb0 0x7f97380762d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 15 client.4324.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 20 client.4324.objecter _get_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 20 client.4324.objecter put_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 20 client.4324.objecter _get_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 15 client.4324.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 20 client.4324.objecter get_session s=0x7f1fe4073790 osd=2 3 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 15 client.4324.objecter _session_command_op_assign 2 2 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 20 client.4324.objecter put_session s=0x7f1fe4073790 osd=2 4 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 10 client.4324.objecter _send_command 2 2024-03-25T02:00:36.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.163+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 --> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1ff00c1fd0 con 0x7f1fe4073e60 2024-03-25T02:00:36.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.170+0000 7f1fdffff640 1 -- 172.21.15.83:0/375716917 <== osd.2 v2:172.21.15.83:6816/602002618 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1fe8037040 con 0x7f1fe4073e60 2024-03-25T02:00:36.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.170+0000 7f1fdffff640 10 client.4324.objecter ms_dispatch 0x7f1ff8000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.170+0000 7f1fdffff640 10 client.4324.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.170+0000 7f1fdffff640 20 client.4324.objecter put_session s=0x7f1fe4073790 osd=2 4 2024-03-25T02:00:36.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.170+0000 7f1fdffff640 15 client.4324.objecter _session_command_op_remove 2 2 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 10 librados: watch_flush enter 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 10 librados: watch_flush exit 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 10 client.4324.objecter close_session for osd.2 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7f1fe4073e60 msgr2=0x7f1fe4076280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7f1fe4073e60 0x7f1fe4076280 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 20 client.4324.objecter put_session s=0x7f1fe4073790 osd=2 1 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.225 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 10 client.4324.objecter successfully canceled tick 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f1fe406da20 msgr2=0x7f1fe406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f1fe406da20 0x7f1fe406fee0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1fec001560 tx=0x7f1fec007990 comp rx=0 tx=0).stop 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 10 monclient: shutdown 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ff00cc630 msgr2=0x7f1ff00c0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 --2- 172.21.15.83:0/375716917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ff00cc630 0x7f1ff00c0e90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ff806ad00 tx=0x7f1ff8068a80 comp rx=0 tx=0).stop 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ff77fe640 1 -- 172.21.15.83:0/375716917 reap_dead start 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 shutdown_connections 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.171+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 >> 172.21.15.83:0/375716917 conn(0x7f1ff0006600 msgr2=0x7f1ff00c3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.226 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.172+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f65b4007400 con 0x7f65b002fc10 2024-03-25T02:00:36.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.172+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 shutdown_connections 2024-03-25T02:00:36.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.172+0000 7f1ffd825640 1 -- 172.21.15.83:0/375716917 wait complete. 2024-03-25T02:00:36.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.172+0000 7f1ffd825640 1 librados: shutdown 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.172+0000 7f65ae7fc640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f659c06da20 0x7f659c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.172+0000 7f65bd796640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f659c06da20 0x7f659c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.172+0000 7f65bd796640 10 monclient: get_auth_request con 0x7f659c06da20 auth_method 0 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.174+0000 7f65ae7fc640 10 client.4336.objecter ms_handle_connect 0x7f659c06da20 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.174+0000 7f65bd796640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f659c06da20 0x7f659c06fee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f65a8000c00 tx=0x7f65a8008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f65b4056fb0 con 0x7f65b002fc10 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 10 client.4336.objecter ms_dispatch 0x7f65b8000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 3 client.4336.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 3 client.4336.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 10 client.4336.objecter checking command 1 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f659c073e60 0x7f659c076280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 20 client.4336.objecter _get_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 20 client.4336.objecter put_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 15 client.4336.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 20 client.4336.objecter _get_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 20 client.4336.objecter get_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 15 client.4336.objecter _session_command_op_assign 3 1 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 20 client.4336.objecter put_session s=0x7f659c073790 osd=3 4 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 10 client.4336.objecter _send_command 1 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 --> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65b4009e50 con 0x7f659c073e60 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f65b40551b0 con 0x7f65b002fc10 2024-03-25T02:00:36.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.179+0000 7f65ae7fc640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.180+0000 7f65be798640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f659c073e60 0x7f659c076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f65be798640 10 monclient: get_auth_request con 0x7f659c073e60 auth_method 0 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4035040 con 0x7f8eb014b0c0 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: dump: 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.233 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea4003610 con 0x7f8eb014b0c0 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4003a00 con 0x7f8eb014b0c0 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eacff9640 10 monclient: dump: 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eae7fc640 4 set_mon_vals no callback set 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 5 monclient: authenticate success, global_id 4339 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 10 monclient: _renew_subs 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 1 -- 172.21.15.83:0/3299155400 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb012e290 con 0x7f8eb014b0c0 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 10 monclient: _renew_subs 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 1 -- 172.21.15.83:0/3299155400 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb01dbb10 con 0x7f8eb014b0c0 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.181+0000 7f8eb5792640 1 librados: init done 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.182+0000 7f8eb5792640 10 client.4339.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.182+0000 7f8eb5792640 15 client.4339.objecter _session_command_op_assign -1 1 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.182+0000 7f8eb5792640 20 client.4339.objecter _get_session osd=-1 returning homeless 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.182+0000 7f8eb5792640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.182+0000 7f8eb5792640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.182+0000 7f8eb5792640 1 -- 172.21.15.83:0/3299155400 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8eb004f940 con 0x7f8eb014b0c0 2024-03-25T02:00:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.183+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fe66c01a070 con 0x7fe6700c1080 2024-03-25T02:00:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.183+0000 7fe666ffd640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe65806da50 0x7fe65806ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.184+0000 7fe67659c640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe65806da50 0x7fe65806ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.184+0000 7fe67659c640 10 monclient: get_auth_request con 0x7fe65806da50 auth_method 0 2024-03-25T02:00:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.184+0000 7f65be798640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f659c073e60 0x7f659c076280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.237 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.184+0000 7f65ae7fc640 10 client.4336.objecter ms_handle_connect 0x7f659c073e60 2024-03-25T02:00:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.184+0000 7fe67659c640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe65806da50 0x7fe65806ff10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe6700319c0 tx=0x7fe6780ab000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.238 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.184+0000 7fe666ffd640 10 client.4253.objecter ms_handle_connect 0x7fe65806da50 2024-03-25T02:00:36.238 INFO:teuthology.orchestra.run.smithi083.stdout:85899345930 2024-03-25T02:00:36.239 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-25T02:00:36.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.192+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== osd.0 v2:172.21.15.83:6800/2678950219 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc7d4076960 con 0x7fc7d4073e90 2024-03-25T02:00:36.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.192+0000 7fc7cf7fe640 10 client.4330.objecter ms_dispatch 0x7fc7e8000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.192+0000 7fc7cf7fe640 10 client.4330.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.192+0000 7fc7cf7fe640 20 client.4330.objecter put_session s=0x7fc7d40737c0 osd=0 4 2024-03-25T02:00:36.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.192+0000 7fc7cf7fe640 15 client.4330.objecter _session_command_op_remove 0 1 2024-03-25T02:00:36.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.198+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f998004e460 con 0x7f998813cef0 2024-03-25T02:00:36.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.199+0000 7f996f7fe640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f997406da70 0x7f997406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.199+0000 7f9986ffd640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f997406da70 0x7f997406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.199+0000 7f9986ffd640 10 monclient: get_auth_request con 0x7f997406da70 auth_method 0 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.200+0000 7f9986ffd640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f997406da70 0x7f997406ff30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9978000f60 tx=0x7f9978009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.200+0000 7f996f7fe640 10 client.4259.objecter ms_handle_connect 0x7f997406da70 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.202+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== osd.6 v2:172.21.15.153:6802/4205760170 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2c14076960 con 0x7f2c14073e90 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.202+0000 7f2c0ffff640 10 client.4311.objecter ms_dispatch 0x7f2c28000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.202+0000 7f2c0ffff640 10 client.4311.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.202+0000 7f2c0ffff640 20 client.4311.objecter put_session s=0x7f2c140737c0 osd=6 4 2024-03-25T02:00:36.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.202+0000 7f2c0ffff640 15 client.4311.objecter _session_command_op_remove 6 1 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.202+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f8ea4038020 con 0x7f8eb014b0c0 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.203+0000 7f8eacff9640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f8e9c06daa0 0x7f8e9c06ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.203+0000 7f8eaffff640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f8e9c06daa0 0x7f8e9c06ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.203+0000 7f8eaffff640 10 monclient: get_auth_request con 0x7f8e9c06daa0 auth_method 0 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.204+0000 7f8eacff9640 10 client.4339.objecter ms_handle_connect 0x7f8e9c06daa0 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.204+0000 7f8eaffff640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f8e9c06daa0 0x7f8e9c06ff60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0005880 tx=0x7f8ea0009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f8ea40798b0 con 0x7f8eb014b0c0 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 10 client.4339.objecter ms_dispatch 0x7f8eb0000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 3 client.4339.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 3 client.4339.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 10 client.4339.objecter checking command 1 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f8e9c073ee0 0x7f8e9c076300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 20 client.4339.objecter _get_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 20 client.4339.objecter put_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 15 client.4339.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 20 client.4339.objecter _get_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 20 client.4339.objecter get_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 15 client.4339.objecter _session_command_op_assign 5 1 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 20 client.4339.objecter put_session s=0x7f8e9c073810 osd=5 4 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 10 client.4339.objecter _send_command 1 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 --> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e9c0769b0 con 0x7f8e9c073ee0 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8ea4077ab0 con 0x7f8eb014b0c0 2024-03-25T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.206+0000 7f8eacff9640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.207+0000 7f8eb4f91640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f8e9c073ee0 0x7f8e9c076300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.207+0000 7f8eb4f91640 10 monclient: get_auth_request con 0x7f8e9c073ee0 auth_method 0 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.210+0000 7f8eb4f91640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f8e9c073ee0 0x7f8e9c076300 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7f8eacff9640 10 client.4339.objecter ms_handle_connect 0x7f8e9c073ee0 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 15 client.4330.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 20 client.4330.objecter _get_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 20 client.4330.objecter put_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 20 client.4330.objecter _get_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 15 client.4330.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 20 client.4330.objecter get_session s=0x7fc7d40737c0 osd=0 3 2024-03-25T02:00:36.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 15 client.4330.objecter _session_command_op_assign 0 2 2024-03-25T02:00:36.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 20 client.4330.objecter put_session s=0x7fc7d40737c0 osd=0 4 2024-03-25T02:00:36.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 10 client.4330.objecter _send_command 2 2024-03-25T02:00:36.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.211+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 --> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc7e0008620 con 0x7fc7d4073e90 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fe66c056db0 con 0x7fe6700c1080 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 10 client.4253.objecter ms_dispatch 0x7fe678000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 3 client.4253.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 3 client.4253.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 10 client.4253.objecter checking command 1 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7fe658073e90 0x7fe6580762b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 20 client.4253.objecter _get_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.214+0000 7fe666ffd640 20 client.4253.objecter put_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 15 client.4253.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 20 client.4253.objecter _get_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 20 client.4253.objecter get_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 15 client.4253.objecter _session_command_op_assign 7 1 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 20 client.4253.objecter put_session s=0x7fe6580737c0 osd=7 4 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 10 client.4253.objecter _send_command 1 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe658076960 con 0x7fe658073e90 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe66c054fb0 con 0x7fe6700c1080 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe666ffd640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.215+0000 7fe675d9b640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7fe658073e90 0x7fe6580762b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f998007f200 con 0x7f998813cef0 2024-03-25T02:00:36.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 10 client.4259.objecter ms_dispatch 0x7f9988000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 3 client.4259.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 3 client.4259.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 10 client.4259.objecter checking command 1 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9974073eb0 0x7f99740762d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 20 client.4259.objecter _get_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 20 client.4259.objecter put_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 15 client.4259.objecter _session_command_op_remove -1 1 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 20 client.4259.objecter _get_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 20 client.4259.objecter get_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 15 client.4259.objecter _session_command_op_assign 4 1 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 20 client.4259.objecter put_session s=0x7f99740737e0 osd=4 4 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 10 client.4259.objecter _send_command 1 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 --> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f998003ae90 con 0x7f9974073eb0 2024-03-25T02:00:36.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7f996f7fe640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.216+0000 7fe675d9b640 10 monclient: get_auth_request con 0x7fe658073e90 auth_method 0 2024-03-25T02:00:36.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7cf7fe640 1 -- 172.21.15.83:0/3422872661 <== osd.0 v2:172.21.15.83:6800/2678950219 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc7e0008620 con 0x7fc7d4073e90 2024-03-25T02:00:36.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7cf7fe640 10 client.4330.objecter ms_dispatch 0x7fc7e8000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7cf7fe640 10 client.4330.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7cf7fe640 20 client.4330.objecter put_session s=0x7fc7d40737c0 osd=0 4 2024-03-25T02:00:36.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7cf7fe640 15 client.4330.objecter _session_command_op_remove 0 2 2024-03-25T02:00:36.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 10 librados: watch_flush enter 2024-03-25T02:00:36.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 10 librados: watch_flush exit 2024-03-25T02:00:36.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 10 client.4330.objecter close_session for osd.0 2024-03-25T02:00:36.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7fc7d4073e90 msgr2=0x7fc7d40762b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7fc7d4073e90 0x7fc7d40762b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 20 client.4330.objecter put_session s=0x7fc7d40737c0 osd=0 1 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 10 client.4330.objecter successfully canceled tick 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fc7d406da50 msgr2=0x7fc7d406ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fc7d406da50 0x7fc7d406ff10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000e170 tx=0x7fc7dc008040 comp rx=0 tx=0).stop 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 10 monclient: shutdown 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7e0038da0 msgr2=0x7fc7e0164b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 --2- 172.21.15.83:0/3422872661 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc7e0038da0 0x7fc7e0164b40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8033000 tx=0x7fc7d80046c0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e77fe640 1 -- 172.21.15.83:0/3422872661 reap_dead start 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 shutdown_connections 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.218+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 >> 172.21.15.83:0/3422872661 conn(0x7fc7e0006600 msgr2=0x7fc7e00384e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.219+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 shutdown_connections 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.219+0000 7fc7e7fff640 1 -- 172.21.15.83:0/3422872661 wait complete. 2024-03-25T02:00:36.253 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.219+0000 7fc7e7fff640 1 librados: shutdown 2024-03-25T02:00:36.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.221+0000 7f99867fc640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9974073eb0 0x7f99740762d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.221+0000 7f99867fc640 10 monclient: get_auth_request con 0x7f9974073eb0 auth_method 0 2024-03-25T02:00:36.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.222+0000 7fe675d9b640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7fe658073e90 0x7fe6580762b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.254 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.222+0000 7fe666ffd640 10 client.4253.objecter ms_handle_connect 0x7fe658073e90 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.223+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== osd.1 v2:172.21.15.83:6824/2597989034 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9744033ec0 con 0x7f9738073eb0 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.223+0000 7f9749ffb640 10 client.4256.objecter ms_dispatch 0x7f9754000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.223+0000 7f9749ffb640 10 client.4256.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.223+0000 7f9749ffb640 20 client.4256.objecter put_session s=0x7f97380737e0 osd=1 4 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.223+0000 7f9749ffb640 15 client.4256.objecter _session_command_op_remove 1 1 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 10 client.4311.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 15 client.4311.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 20 client.4311.objecter _get_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 20 client.4311.objecter put_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 20 client.4311.objecter _get_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 15 client.4311.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 20 client.4311.objecter get_session s=0x7f2c140737c0 osd=6 3 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 15 client.4311.objecter _session_command_op_assign 6 2 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 20 client.4311.objecter put_session s=0x7f2c140737c0 osd=6 4 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 10 client.4311.objecter _send_command 2 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.231+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 --> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2c200052f0 con 0x7f2c14073e90 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.233+0000 7f996f7fe640 10 client.4259.objecter ms_handle_connect 0x7f9974073eb0 2024-03-25T02:00:36.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.233+0000 7f99867fc640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9974073eb0 0x7f99740762d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.257 INFO:teuthology.orchestra.run.smithi083.stdout:64424509447 2024-03-25T02:00:36.257 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-25T02:00:36.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c0ffff640 1 -- 172.21.15.83:0/85110343 <== osd.6 v2:172.21.15.153:6802/4205760170 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2c200052f0 con 0x7f2c14073e90 2024-03-25T02:00:36.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c0ffff640 10 client.4311.objecter ms_dispatch 0x7f2c28000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c0ffff640 10 client.4311.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c0ffff640 20 client.4311.objecter put_session s=0x7f2c140737c0 osd=6 4 2024-03-25T02:00:36.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c0ffff640 15 client.4311.objecter _session_command_op_remove 6 2 2024-03-25T02:00:36.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 10 librados: watch_flush enter 2024-03-25T02:00:36.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 10 librados: watch_flush exit 2024-03-25T02:00:36.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 10 client.4311.objecter close_session for osd.6 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7f2c14073e90 msgr2=0x7f2c140762b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7f2c14073e90 0x7f2c140762b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 20 client.4311.objecter put_session s=0x7f2c140737c0 osd=6 1 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.238+0000 7f2c2da45640 10 client.4311.objecter successfully canceled tick 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2c1406da50 msgr2=0x7f2c1406ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2c1406da50 0x7f2c1406ff10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0090b0 tx=0x7f2c1c03c040 comp rx=0 tx=0).stop 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 10 monclient: shutdown 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c20006a30 msgr2=0x7f2c200c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 --2- 172.21.15.83:0/85110343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c20006a30 0x7f2c200c7370 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2c28050bb0 tx=0x7f2c28068a80 comp rx=0 tx=0).stop 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c277fe640 1 -- 172.21.15.83:0/85110343 reap_dead start 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 shutdown_connections 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 >> 172.21.15.83:0/85110343 conn(0x7f2c20005f10 msgr2=0x7f2c200c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 shutdown_connections 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 -- 172.21.15.83:0/85110343 wait complete. 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.239+0000 7f2c2da45640 1 librados: shutdown 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.244+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== osd.3 v2:172.21.15.83:6808/3432501411 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f65b4009e50 con 0x7f659c073e60 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.244+0000 7f65ae7fc640 10 client.4336.objecter ms_dispatch 0x7f65b8000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.244+0000 7f65ae7fc640 10 client.4336.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.244+0000 7f65ae7fc640 20 client.4336.objecter put_session s=0x7f659c073790 osd=3 4 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.244+0000 7f65ae7fc640 15 client.4336.objecter _session_command_op_remove 3 1 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 15 client.4256.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 20 client.4256.objecter _get_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 20 client.4256.objecter put_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 20 client.4256.objecter _get_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 15 client.4256.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 20 client.4256.objecter get_session s=0x7f97380737e0 osd=1 3 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 15 client.4256.objecter _session_command_op_assign 1 2 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 20 client.4256.objecter put_session s=0x7f97380737e0 osd=1 4 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 10 client.4256.objecter _send_command 2 2024-03-25T02:00:36.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.252+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 --> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f975404f940 con 0x7f9738073eb0 2024-03-25T02:00:36.263 INFO:teuthology.orchestra.run.smithi083.stdout:81604378634 2024-03-25T02:00:36.263 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 10 client.4336.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 15 client.4336.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 20 client.4336.objecter _get_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 20 client.4336.objecter put_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 20 client.4336.objecter _get_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 15 client.4336.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 20 client.4336.objecter get_session s=0x7f659c073790 osd=3 3 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 15 client.4336.objecter _session_command_op_assign 3 2 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 20 client.4336.objecter put_session s=0x7f659c073790 osd=3 4 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 10 client.4336.objecter _send_command 2 2024-03-25T02:00:36.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.260+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 --> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f65a0001240 con 0x7f659c073e60 2024-03-25T02:00:36.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f9749ffb640 1 -- 172.21.15.83:0/2545748317 <== osd.1 v2:172.21.15.83:6824/2597989034 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f975404f940 con 0x7f9738073eb0 2024-03-25T02:00:36.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f9749ffb640 10 client.4256.objecter ms_dispatch 0x7f9754000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f9749ffb640 10 client.4256.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f9749ffb640 20 client.4256.objecter put_session s=0x7f97380737e0 osd=1 4 2024-03-25T02:00:36.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f9749ffb640 15 client.4256.objecter _session_command_op_remove 1 2 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 10 librados: watch_flush enter 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 10 librados: watch_flush exit 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 10 client.4256.objecter close_session for osd.1 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f9738073eb0 msgr2=0x7f97380762d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f9738073eb0 0x7f97380762d0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 20 client.4256.objecter put_session s=0x7f97380737e0 osd=1 1 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.261+0000 7f975b4e4640 10 client.4256.objecter successfully canceled tick 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f973806da70 msgr2=0x7f973806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f973806da70 0x7f973806ff30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f97500015f0 tx=0x7f9750007a40 comp rx=0 tx=0).stop 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 10 monclient: shutdown 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f97541446b0 msgr2=0x7f975414aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 --2- 172.21.15.83:0/2545748317 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f97541446b0 0x7f975414aa70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f974402beb0 tx=0x7f97440046c0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f9759a81640 1 -- 172.21.15.83:0/2545748317 reap_dead start 2024-03-25T02:00:36.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 shutdown_connections 2024-03-25T02:00:36.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 >> 172.21.15.83:0/2545748317 conn(0x7f975406d0f0 msgr2=0x7f97540b2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 shutdown_connections 2024-03-25T02:00:36.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 -- 172.21.15.83:0/2545748317 wait complete. 2024-03-25T02:00:36.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.262+0000 7f975b4e4640 1 librados: shutdown 2024-03-25T02:00:36.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.269+0000 7f65ae7fc640 1 -- 172.21.15.83:0/2352701376 <== osd.3 v2:172.21.15.83:6808/3432501411 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f65a0001240 con 0x7f659c073e60 2024-03-25T02:00:36.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.269+0000 7f65ae7fc640 10 client.4336.objecter ms_dispatch 0x7f65b8000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.269+0000 7f65ae7fc640 10 client.4336.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.269+0000 7f65ae7fc640 20 client.4336.objecter put_session s=0x7f659c073790 osd=3 4 2024-03-25T02:00:36.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.269+0000 7f65ae7fc640 15 client.4336.objecter _session_command_op_remove 3 2 2024-03-25T02:00:36.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 10 librados: watch_flush enter 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 10 librados: watch_flush exit 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 10 client.4336.objecter close_session for osd.3 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f659c073e60 msgr2=0x7f659c076280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f659c073e60 0x7f659c076280 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 20 client.4336.objecter put_session s=0x7f659c073790 osd=3 1 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 10 client.4336.objecter successfully canceled tick 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f659c06da20 msgr2=0x7f659c06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f659c06da20 0x7f659c06fee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f65a8000c00 tx=0x7f65a8008030 comp rx=0 tx=0).stop 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 10 monclient: shutdown 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65b002fc10 msgr2=0x7f65b00c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.270+0000 7f65bef99640 1 --2- 172.21.15.83:0/2352701376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65b002fc10 0x7f65b00c4940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65b400c340 tx=0x7f65b400c800 comp rx=0 tx=0).stop 2024-03-25T02:00:36.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.271+0000 7f65be798640 1 -- 172.21.15.83:0/2352701376 reap_dead start 2024-03-25T02:00:36.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.271+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 shutdown_connections 2024-03-25T02:00:36.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.271+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 >> 172.21.15.83:0/2352701376 conn(0x7f65b00060c0 msgr2=0x7f65b00cfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.271+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 shutdown_connections 2024-03-25T02:00:36.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.271+0000 7f65bef99640 1 -- 172.21.15.83:0/2352701376 wait complete. 2024-03-25T02:00:36.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.271+0000 7f65bef99640 1 librados: shutdown 2024-03-25T02:00:36.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.275+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== osd.5 v2:172.21.15.153:6810/2458137144 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e9c0769b0 con 0x7f8e9c073ee0 2024-03-25T02:00:36.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.275+0000 7f8eacff9640 10 client.4339.objecter ms_dispatch 0x7f8eb0000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.275+0000 7f8eacff9640 10 client.4339.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.275+0000 7f8eacff9640 20 client.4339.objecter put_session s=0x7f8e9c073810 osd=5 4 2024-03-25T02:00:36.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.275+0000 7f8eacff9640 15 client.4339.objecter _session_command_op_remove 5 1 2024-03-25T02:00:36.279 INFO:teuthology.orchestra.run.smithi083.stdout:85899345931 2024-03-25T02:00:36.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-25T02:00:36.290 INFO:teuthology.orchestra.run.smithi083.stdout:64424509447 2024-03-25T02:00:36.290 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-25T02:00:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.296+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== osd.7 v2:172.21.15.153:6826/2580901001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe658076960 con 0x7fe658073e90 2024-03-25T02:00:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.296+0000 7fe666ffd640 10 client.4253.objecter ms_dispatch 0x7fe678000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.296+0000 7fe666ffd640 10 client.4253.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.296+0000 7fe666ffd640 20 client.4253.objecter put_session s=0x7fe6580737c0 osd=7 4 2024-03-25T02:00:36.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.296+0000 7fe666ffd640 15 client.4253.objecter _session_command_op_remove 7 1 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 10 client.4339.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 15 client.4339.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 20 client.4339.objecter _get_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 20 client.4339.objecter put_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 20 client.4339.objecter _get_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 15 client.4339.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 20 client.4339.objecter get_session s=0x7f8e9c073810 osd=5 3 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 15 client.4339.objecter _session_command_op_assign 5 2 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 20 client.4339.objecter put_session s=0x7f8e9c073810 osd=5 4 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 10 client.4339.objecter _send_command 2 2024-03-25T02:00:36.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.297+0000 7f8eb5792640 1 -- 172.21.15.83:0/3299155400 --> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8eb004f940 con 0x7f8e9c073ee0 2024-03-25T02:00:36.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.311+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== osd.4 v2:172.21.15.153:6818/2579144563 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f998003ae90 con 0x7f9974073eb0 2024-03-25T02:00:36.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.311+0000 7f996f7fe640 10 client.4259.objecter ms_dispatch 0x7f9988000b90 command_reply(tid 1: 0 ) 2024-03-25T02:00:36.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.311+0000 7f996f7fe640 10 client.4259.objecter _finish_command 1 = system:0 2024-03-25T02:00:36.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.311+0000 7f996f7fe640 20 client.4259.objecter put_session s=0x7f99740737e0 osd=4 4 2024-03-25T02:00:36.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.311+0000 7f996f7fe640 15 client.4259.objecter _session_command_op_remove 4 1 2024-03-25T02:00:36.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.317+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f998007d400 con 0x7f998813cef0 2024-03-25T02:00:36.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.317+0000 7f996f7fe640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:00:36.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.319+0000 7f8eacff9640 1 -- 172.21.15.83:0/3299155400 <== osd.5 v2:172.21.15.153:6810/2458137144 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8eb004f940 con 0x7f8e9c073ee0 2024-03-25T02:00:36.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.319+0000 7f8eacff9640 10 client.4339.objecter ms_dispatch 0x7f8eb0000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.319+0000 7f8eacff9640 10 client.4339.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.319+0000 7f8eacff9640 20 client.4339.objecter put_session s=0x7f8e9c073810 osd=5 4 2024-03-25T02:00:36.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.319+0000 7f8eacff9640 15 client.4339.objecter _session_command_op_remove 5 2 2024-03-25T02:00:36.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 10 librados: watch_flush enter 2024-03-25T02:00:36.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 10 librados: watch_flush exit 2024-03-25T02:00:36.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 10 client.4339.objecter close_session for osd.5 2024-03-25T02:00:36.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 1 -- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f8e9c073ee0 msgr2=0x7f8e9c076300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f8e9c073ee0 0x7f8e9c076300 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 20 client.4339.objecter put_session s=0x7f8e9c073810 osd=5 1 2024-03-25T02:00:36.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.320+0000 7f8e967fc640 10 client.4339.objecter successfully canceled tick 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 1 -- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f8e9c06daa0 msgr2=0x7f8e9c06ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f8e9c06daa0 0x7f8e9c06ff60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0005880 tx=0x7f8ea0009040 comp rx=0 tx=0).stop 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 10 monclient: shutdown 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 1 -- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb014b0c0 msgr2=0x7f8eb014b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 1 --2- 172.21.15.83:0/3299155400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8eb014b0c0 0x7f8eb014b530 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8eb006c650 tx=0x7f8ea40046c0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8eb4f91640 1 -- 172.21.15.83:0/3299155400 reap_dead start 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 1 -- 172.21.15.83:0/3299155400 shutdown_connections 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.321+0000 7f8e967fc640 1 -- 172.21.15.83:0/3299155400 >> 172.21.15.83:0/3299155400 conn(0x7f8eb00b4b80 msgr2=0x7f8eb00b4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.322+0000 7f8e967fc640 1 -- 172.21.15.83:0/3299155400 shutdown_connections 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.322+0000 7f8e967fc640 1 -- 172.21.15.83:0/3299155400 wait complete. 2024-03-25T02:00:36.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.322+0000 7f8e967fc640 1 librados: shutdown 2024-03-25T02:00:36.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 10 client.4253.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 15 client.4253.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 20 client.4253.objecter _get_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 20 client.4253.objecter put_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 20 client.4253.objecter _get_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 15 client.4253.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 20 client.4253.objecter get_session s=0x7fe6580737c0 osd=7 3 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 15 client.4253.objecter _session_command_op_assign 7 2 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 20 client.4253.objecter put_session s=0x7fe6580737c0 osd=7 4 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 10 client.4253.objecter _send_command 2 2024-03-25T02:00:36.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.328+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe65c001240 con 0x7fe658073e90 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.338+0000 7fe666ffd640 1 -- 172.21.15.83:0/2725632370 <== osd.7 v2:172.21.15.153:6826/2580901001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe65c001240 con 0x7fe658073e90 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.338+0000 7fe666ffd640 10 client.4253.objecter ms_dispatch 0x7fe678000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.338+0000 7fe666ffd640 10 client.4253.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.338+0000 7fe666ffd640 20 client.4253.objecter put_session s=0x7fe6580737c0 osd=7 4 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.338+0000 7fe666ffd640 15 client.4253.objecter _session_command_op_remove 7 2 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.338+0000 7fe67cdaf640 10 librados: watch_flush enter 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 10 librados: watch_flush exit 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 10 client.4253.objecter close_session for osd.7 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7fe658073e90 msgr2=0x7fe6580762b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7fe658073e90 0x7fe6580762b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 20 client.4253.objecter put_session s=0x7fe6580737c0 osd=7 1 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 10 client.4253.objecter successfully canceled tick 2024-03-25T02:00:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe65806da50 msgr2=0x7fe65806ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe65806da50 0x7fe65806ff10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe6700319c0 tx=0x7fe6780ab000 comp rx=0 tx=0).stop 2024-03-25T02:00:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 10 monclient: shutdown 2024-03-25T02:00:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe6700c1080 msgr2=0x7fe6700d4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe67cdaf640 1 --2- 172.21.15.83:0/2725632370 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe6700c1080 0x7fe6700d4cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00c390 tx=0x7fe66c00c850 comp rx=0 tx=0).stop 2024-03-25T02:00:36.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.339+0000 7fe676d9d640 1 -- 172.21.15.83:0/2725632370 reap_dead start 2024-03-25T02:00:36.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.341+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 shutdown_connections 2024-03-25T02:00:36.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.341+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 >> 172.21.15.83:0/2725632370 conn(0x7fe670005ed0 msgr2=0x7fe670036e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.341+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 shutdown_connections 2024-03-25T02:00:36.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.341+0000 7fe67cdaf640 1 -- 172.21.15.83:0/2725632370 wait complete. 2024-03-25T02:00:36.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.341+0000 7fe67cdaf640 1 librados: shutdown 2024-03-25T02:00:36.344 INFO:teuthology.orchestra.run.smithi083.stdout:77309411334 2024-03-25T02:00:36.345 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-25T02:00:36.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 10 client.4259.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 15 client.4259.objecter _session_command_op_assign -1 2 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 20 client.4259.objecter _get_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 20 client.4259.objecter put_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 20 client.4259.objecter _get_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 15 client.4259.objecter _session_command_op_remove -1 2 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 20 client.4259.objecter get_session s=0x7f99740737e0 osd=4 3 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 15 client.4259.objecter _session_command_op_assign 4 2 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 20 client.4259.objecter put_session s=0x7f99740737e0 osd=4 4 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 10 client.4259.objecter _send_command 2 2024-03-25T02:00:36.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.342+0000 7f998d359640 1 -- 172.21.15.83:0/3177696750 --> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f998804f940 con 0x7f9974073eb0 2024-03-25T02:00:36.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.349+0000 7f996f7fe640 1 -- 172.21.15.83:0/3177696750 <== osd.4 v2:172.21.15.153:6818/2579144563 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f998804f940 con 0x7f9974073eb0 2024-03-25T02:00:36.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.349+0000 7f996f7fe640 10 client.4259.objecter ms_dispatch 0x7f9988000b90 command_reply(tid 2: 0 ) 2024-03-25T02:00:36.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.349+0000 7f996f7fe640 10 client.4259.objecter _finish_command 2 = system:0 2024-03-25T02:00:36.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.349+0000 7f996f7fe640 20 client.4259.objecter put_session s=0x7f99740737e0 osd=4 4 2024-03-25T02:00:36.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.349+0000 7f996f7fe640 15 client.4259.objecter _session_command_op_remove 4 2 2024-03-25T02:00:36.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 10 librados: watch_flush enter 2024-03-25T02:00:36.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 10 librados: watch_flush exit 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 10 client.4259.objecter close_session for osd.4 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 -- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9974073eb0 msgr2=0x7f99740762d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9974073eb0 0x7f99740762d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 20 client.4259.objecter put_session s=0x7f99740737e0 osd=4 1 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 10 client.4259.objecter successfully canceled tick 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 -- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f997406da70 msgr2=0x7f997406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f997406da70 0x7f997406ff30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9978000f60 tx=0x7f9978009040 comp rx=0 tx=0).stop 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 10 monclient: shutdown 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 -- 172.21.15.83:0/3177696750 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f998813cef0 msgr2=0x7f99881d7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 --2- 172.21.15.83:0/3177696750 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f998813cef0 0x7f99881d7890 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f998003b040 tx=0x7f9980004a70 comp rx=0 tx=0).stop 2024-03-25T02:00:36.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f9987fff640 1 -- 172.21.15.83:0/3177696750 reap_dead start 2024-03-25T02:00:36.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 -- 172.21.15.83:0/3177696750 shutdown_connections 2024-03-25T02:00:36.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.350+0000 7f996d7fa640 1 -- 172.21.15.83:0/3177696750 >> 172.21.15.83:0/3177696750 conn(0x7f998806b7e0 msgr2=0x7f99880b3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.351+0000 7f996d7fa640 1 -- 172.21.15.83:0/3177696750 shutdown_connections 2024-03-25T02:00:36.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.351+0000 7f996d7fa640 1 -- 172.21.15.83:0/3177696750 wait complete. 2024-03-25T02:00:36.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.351+0000 7f996d7fa640 1 librados: shutdown 2024-03-25T02:00:36.374 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-03-25T02:00:36.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-25T02:00:36.378 INFO:teuthology.orchestra.run.smithi083.stdout:81604378632 2024-03-25T02:00:36.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-25T02:00:36.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.410+0000 7f2927daf640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.410+0000 7f2927daf640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.410+0000 7f2927daf640 10 monclient: monmap: 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.412154+0000 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.412154+0000 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.413 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.411+0000 7f2927daf640 1 Processor -- start 2024-03-25T02:00:36.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 1 -- start start 2024-03-25T02:00:36.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient: init 2024-03-25T02:00:36.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.415 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient(hunting): picked mon.noname-b con 0x7f29200b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f29200b2df0 0x7f2920147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient(hunting): picked mon.noname-c con 0x7f29200b2df0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f2920140400 0x7f2920140870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient(hunting): picked mon.noname-a con 0x7f2920140400 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f29200b2410 con 0x7f2920140400 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.416 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f29200b26f0 con 0x7f29200b2df0 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f29200b29d0 con 0x7f29200b04a0 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.412+0000 7f2927daf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.413+0000 7f2927daf640 10 monclient(hunting): authenticate will time out at 1355.602962s 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.413+0000 7f2925b4b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.414+0000 7f292534a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f29200b2df0 0x7f2920147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.414+0000 7f292534a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f29200b2df0 0x7f2920147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40578/0 (socket says 172.21.15.83:40578) 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.414+0000 7f292534a640 1 -- 172.21.15.83:0/387045405 learned_addr learned my addr 172.21.15.83:0/387045405 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.414+0000 7f292534a640 10 monclient(hunting): get_auth_request con 0x7f29200b2df0 auth_method 0 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.414+0000 7f292534a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.414+0000 7f292534a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.414+0000 7f292534a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f2925b4b640 10 monclient(hunting): get_auth_request con 0x7f29200b04a0 auth_method 0 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f2925b4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f2925b4b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f2925b4b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292534a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292534a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292534a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292634c640 1 --2- 172.21.15.83:0/387045405 >> v2:172.21.15.83:3300/0 conn(0x7f2920140400 0x7f2920140870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292634c640 10 monclient(hunting): get_auth_request con 0x7f2920140400 auth_method 0 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292634c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292634c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.415+0000 7f292634c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.416+0000 7f2925b4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.416+0000 7f2925b4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.416+0000 7f2925b4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.418+0000 7f292634c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.418+0000 7f292634c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.418+0000 7f292634c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.421+0000 7f2925b4b640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-03-25T02:00:36.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.421+0000 7f2925b4b640 1 -- 172.21.15.83:0/387045405 >> v2:172.21.15.153:3300/0 conn(0x7f29200b2df0 msgr2=0x7f2920147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.421+0000 7f2925b4b640 1 --2- 172.21.15.83:0/387045405 >> v2:172.21.15.153:3300/0 conn(0x7f29200b2df0 0x7f2920147180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.421+0000 7f2925b4b640 1 -- 172.21.15.83:0/387045405 >> v2:172.21.15.83:3300/0 conn(0x7f2920140400 msgr2=0x7f2920140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.422+0000 7f2925b4b640 1 --2- 172.21.15.83:0/387045405 >> v2:172.21.15.83:3300/0 conn(0x7f2920140400 0x7f2920140870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.422+0000 7f2925b4b640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.422+0000 7f2925b4b640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.422+0000 7f2925b4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.422+0000 7f2925b4b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.422+0000 7f2925b4b640 1 -- 172.21.15.83:0/387045405 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2920140db0 con 0x7f29200b04a0 2024-03-25T02:00:36.426 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.422+0000 7f292634c640 1 --2- 172.21.15.83:0/387045405 >> v2:172.21.15.83:3300/0 conn(0x7f2920140400 0x7f2920140870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T02:00:36.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.425+0000 7f2925b4b640 1 --2- 172.21.15.83:0/387045405 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920146c40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f29200b6570 tx=0x7f291802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a4cfff8a13fdc9d server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.428+0000 7f2924b49640 1 -- 172.21.15.83:0/387045405 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918039070 con 0x7f29200b04a0 2024-03-25T02:00:36.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.428+0000 7f2924b49640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.428+0000 7f2924b49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.428+0000 7f2924b49640 10 monclient: dump: 2024-03-25T02:00:36.430 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.430 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.428+0000 7f2924b49640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.428+0000 7f2924b49640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.428+0000 7f2924b49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.429+0000 7f2927daf640 5 monclient: authenticate success, global_id 4268 2024-03-25T02:00:36.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.429+0000 7f2927daf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:36.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.429+0000 7f622f8b6640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.429+0000 7f622f8b6640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.429+0000 7f622f8b6640 10 monclient: monmap: 2024-03-25T02:00:36.432 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.431119+0000 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.431119+0000 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.430+0000 7f622f8b6640 1 Processor -- start 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.430+0000 7f622f8b6640 1 -- start start 2024-03-25T02:00:36.433 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.430+0000 7f622f8b6640 10 monclient: init 2024-03-25T02:00:36.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2924b49640 1 -- 172.21.15.83:0/387045405 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f291802cc60 con 0x7f29200b04a0 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2924b49640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2924b49640 1 -- 172.21.15.83:0/387045405 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918034660 con 0x7f29200b04a0 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2924b49640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2924b49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2924b49640 10 monclient: dump: 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.435 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-03-25T02:00:36.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2927daf640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.432+0000 7f2927daf640 4 set_mon_vals no callback set 2024-03-25T02:00:36.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 10 monclient: shutdown 2024-03-25T02:00:36.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 1 -- 172.21.15.83:0/387045405 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 msgr2=0x7f2920146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 1 --2- 172.21.15.83:0/387045405 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920146c40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f29200b6570 tx=0x7f291802beb0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6228071c60 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f622813e6b0 0x7f6228141350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient(hunting): picked mon.noname-c con 0x7f622813e6b0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6228141890 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62280a9a10 con 0x7f6228071c60 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62280a9cf0 con 0x7f6228141890 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f62280b13c0 con 0x7f622813e6b0 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.431+0000 7f622f8b6640 10 monclient(hunting): authenticate will time out at 1355.621914s 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f622de53640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6228141890 0x7f622814b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33322/0 (socket says 172.21.15.83:33322) 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f622de53640 1 -- 172.21.15.83:0/3456767378 learned_addr learned my addr 172.21.15.83:0/3456767378 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f622ce51640 1 --2- 172.21.15.83:0/3456767378 >> v2:172.21.15.153:3300/0 conn(0x7f622813e6b0 0x7f6228141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f622ce51640 10 monclient(hunting): get_auth_request con 0x7f622813e6b0 auth_method 0 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f622ce51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f622ce51640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f622ce51640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 1 -- 172.21.15.83:0/387045405 shutdown_connections 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 1 --2- 172.21.15.83:0/387045405 >> v2:172.21.15.83:3300/0 conn(0x7f2920140400 0x7f2920140870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 1 --2- 172.21.15.83:0/387045405 >> v2:172.21.15.153:3300/0 conn(0x7f29200b2df0 0x7f2920147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 1 --2- 172.21.15.83:0/387045405 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920146c40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.433+0000 7f2927daf640 1 -- 172.21.15.83:0/387045405 >> 172.21.15.83:0/387045405 conn(0x7f29200b8a00 msgr2=0x7f29200b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.434+0000 7f2927daf640 1 -- 172.21.15.83:0/387045405 shutdown_connections 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.435+0000 7f2927daf640 1 -- 172.21.15.83:0/387045405 wait complete. 2024-03-25T02:00:36.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.435+0000 7f2927daf640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.435+0000 7f2927daf640 10 monclient: monmap: 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.436+0000 7f2927daf640 1 librados: starting msgr at 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.436+0000 7f2927daf640 1 librados: starting objecter 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.436+0000 7f2927daf640 1 Processor -- start 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.436+0000 7f2927daf640 1 -- start start 2024-03-25T02:00:36.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.436+0000 7f2927daf640 1 librados: setting wanted keys 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.436+0000 7f2927daf640 1 librados: calling monclient init 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.436+0000 7f2927daf640 10 monclient: init 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920145d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): picked mon.noname-c con 0x7f29200b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29200b2df0 0x7f292006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): picked mon.noname-a con 0x7f29200b2df0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2920140400 0x7f2920070f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): picked mon.noname-b con 0x7f2920140400 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29200b0a00 con 0x7f29200b2df0 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29200b0880 con 0x7f2920140400 2024-03-25T02:00:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2920143200 con 0x7f29200b04a0 2024-03-25T02:00:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.437+0000 7f2927daf640 10 monclient(hunting): authenticate will time out at 1355.627737s 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.434+0000 7f622ce51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.434+0000 7f622ce51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.434+0000 7f622ce51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 1 -- 172.21.15.83:0/3456767378 >> v1:172.21.15.153:6789/0 conn(0x7f6228141890 legacy=0x7f622814b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 1 -- 172.21.15.83:0/3456767378 >> v1:172.21.15.83:6789/0 conn(0x7f6228071c60 legacy=0x7f622813dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.438+0000 7f622ce51640 1 -- 172.21.15.83:0/3456767378 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622806e730 con 0x7f622813e6b0 2024-03-25T02:00:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f2925b4b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f2925b4b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920145d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54506/0 (socket says 172.21.15.83:54506) 2024-03-25T02:00:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f2925b4b640 1 -- 172.21.15.83:0/367446305 learned_addr learned my addr 172.21.15.83:0/367446305 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f2925b4b640 10 monclient(hunting): get_auth_request con 0x7f29200b04a0 auth_method 0 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f2925b4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f2925b4b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f2925b4b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f292534a640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29200b2df0 0x7f292006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f292534a640 10 monclient(hunting): get_auth_request con 0x7f29200b2df0 auth_method 0 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f292534a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f292534a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f292534a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f292634c640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2920140400 0x7f2920070f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f7432dcf640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f7432dcf640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.439+0000 7f7432dcf640 10 monclient: monmap: 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.441166+0000 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.441166+0000 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.447 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.440+0000 7f2925b4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.440+0000 7f2925b4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.440+0000 7f2925b4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.440+0000 7f292634c640 10 monclient(hunting): get_auth_request con 0x7f2920140400 auth_method 0 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.440+0000 7f292634c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.440+0000 7f292634c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.440+0000 7f292634c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f292634c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f292634c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f292634c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.441+0000 7f7432dcf640 1 Processor -- start 2024-03-25T02:00:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.441+0000 7f7432dcf640 1 -- start start 2024-03-25T02:00:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.441+0000 7f7432dcf640 10 monclient: init 2024-03-25T02:00:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.450 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f7424034c10 0x7f74240cc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f7424034c10 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f742402fc90 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7424032150 0x7f74240c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7424032150 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7424033230 con 0x7f742402fc90 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f7424034ff0 con 0x7f7424034c10 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f7424031c30 con 0x7f7424032150 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.442+0000 7f7432dcf640 10 monclient(hunting): authenticate will time out at 1355.632318s 2024-03-25T02:00:36.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.443+0000 7f292534a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.443+0000 7f292534a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.443+0000 7f292534a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.443+0000 7f74315cc640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f742402fc90 0x7f74240c91b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45670/0 (socket says 172.21.15.83:45670) 2024-03-25T02:00:36.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.443+0000 7f74315cc640 1 -- 172.21.15.83:0/2212337095 learned_addr learned my addr 172.21.15.83:0/2212337095 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-03-25T02:00:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 1 -- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2920140400 msgr2=0x7f2920070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2920140400 0x7f2920070f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 1 -- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29200b2df0 msgr2=0x7f292006ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29200b2df0 0x7f292006ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.445+0000 7f2925b4b640 1 -- 172.21.15.83:0/367446305 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2920070540 con 0x7f29200b04a0 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.447+0000 7f7431dcd640 1 --2- 172.21.15.83:0/2212337095 >> v2:172.21.15.83:3300/0 conn(0x7f7424034c10 0x7f74240cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.447+0000 7f74325ce640 1 --2- 172.21.15.83:0/2212337095 >> v2:172.21.15.153:3300/0 conn(0x7f7424032150 0x7f74240c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.447+0000 7f74325ce640 10 monclient(hunting): get_auth_request con 0x7f7424032150 auth_method 0 2024-03-25T02:00:36.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.447+0000 7f74325ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.447+0000 7f74325ce640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.447+0000 7f74325ce640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.448+0000 7f622ce51640 1 --2- 172.21.15.83:0/3456767378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 0x7f6228141350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f62240097a0 tx=0x7f622402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f620ddc1f6fe706 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.448+0000 7f2925b4b640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920145d90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2918009e80 tx=0x7f2918004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.448+0000 7f29167fc640 10 client.?.objecter ms_handle_connect 0x7f29200b04a0 2024-03-25T02:00:36.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.448+0000 7f29167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.448+0000 7f74325ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.448+0000 7f74325ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.448+0000 7f74325ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.450+0000 7f74325ce640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.450+0000 7f74325ce640 1 -- 172.21.15.83:0/2212337095 >> v2:172.21.15.83:3300/0 conn(0x7f7424034c10 msgr2=0x7f74240cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.451+0000 7f74325ce640 1 --2- 172.21.15.83:0/2212337095 >> v2:172.21.15.83:3300/0 conn(0x7f7424034c10 0x7f74240cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.451+0000 7f74325ce640 1 -- 172.21.15.83:0/2212337095 >> v1:172.21.15.83:6789/0 conn(0x7f742402fc90 legacy=0x7f74240c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.451+0000 7f74325ce640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.451+0000 7f74325ce640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.451+0000 7f74325ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.451+0000 7f74325ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.451+0000 7f74325ce640 1 -- 172.21.15.83:0/2212337095 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74240316c0 con 0x7f7424032150 2024-03-25T02:00:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f29167fc640 1 -- 172.21.15.83:0/367446305 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918039050 con 0x7f29200b04a0 2024-03-25T02:00:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f29167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f29167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f29167fc640 10 monclient: dump: 2024-03-25T02:00:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f29167fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f29167fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f29167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 5 monclient: authenticate success, global_id 4271 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 10 monclient: _renew_subs 2024-03-25T02:00:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2920070820 con 0x7f29200b04a0 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 10 monclient: _renew_subs 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f292013bb00 con 0x7f29200b04a0 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.452+0000 7f2927daf640 1 librados: init done 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f29167fc640 1 -- 172.21.15.83:0/367446305 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2918045070 con 0x7f29200b04a0 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f29167fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f29167fc640 1 -- 172.21.15.83:0/367446305 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918009450 con 0x7f29200b04a0 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f29167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f29167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f29167fc640 10 monclient: dump: 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.462 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f2927daf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f2927daf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f2927daf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.454+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29200b7bb0 con 0x7f29200b04a0 2024-03-25T02:00:36.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.455+0000 7f29177fe640 4 set_mon_vals no callback set 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.455+0000 7f621ffff640 1 -- 172.21.15.83:0/3456767378 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6224035020 con 0x7f622813e6b0 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.455+0000 7f621ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.455+0000 7f621ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.455+0000 7f621ffff640 10 monclient: dump: 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.464 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f621ffff640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f621ffff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f621ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 5 monclient: authenticate success, global_id 4345 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f621ffff640 1 -- 172.21.15.83:0/3456767378 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6224031640 con 0x7f622813e6b0 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f621ffff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 4 set_mon_vals no callback set 2024-03-25T02:00:36.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f74325ce640 1 --2- 172.21.15.83:0/2212337095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240c98c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f742400a3e0 tx=0x7f741c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=913fb137941b0113 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 10 monclient: shutdown 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 1 -- 172.21.15.83:0/3456767378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 msgr2=0x7f6228141350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.456+0000 7f622f8b6640 1 --2- 172.21.15.83:0/3456767378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 0x7f6228141350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f62240097a0 tx=0x7f622402b8f0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.457+0000 7f622f8b6640 1 -- 172.21.15.83:0/3456767378 shutdown_connections 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.457+0000 7f622f8b6640 1 --2- 172.21.15.83:0/3456767378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 0x7f6228141350 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.457+0000 7f622f8b6640 1 -- 172.21.15.83:0/3456767378 >> 172.21.15.83:0/3456767378 conn(0x7f62280b69c0 msgr2=0x7f62280b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.457+0000 7f622f8b6640 1 -- 172.21.15.83:0/3456767378 shutdown_connections 2024-03-25T02:00:36.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.457+0000 7f622f8b6640 1 -- 172.21.15.83:0/3456767378 wait complete. 2024-03-25T02:00:36.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 1 -- 172.21.15.83:0/2212337095 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c0315a0 con 0x7f7424032150 2024-03-25T02:00:36.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: dump: 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 1 -- 172.21.15.83:0/2212337095 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f741c037070 con 0x7f7424032150 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 1 -- 172.21.15.83:0/2212337095 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c031960 con 0x7f7424032150 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7430dcb640 10 monclient: dump: 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.470 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7432dcf640 5 monclient: authenticate success, global_id 4354 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7432dcf640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.460+0000 7f7432dcf640 4 set_mon_vals no callback set 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 10 monclient: shutdown 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 1 -- 172.21.15.83:0/2212337095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 msgr2=0x7f74240c98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2212337095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240c98c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f742400a3e0 tx=0x7f741c02b840 comp rx=0 tx=0).stop 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 1 -- 172.21.15.83:0/2212337095 shutdown_connections 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2212337095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240c98c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2212337095 >> v2:172.21.15.83:3300/0 conn(0x7f7424034c10 0x7f74240cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.461+0000 7f7432dcf640 1 -- 172.21.15.83:0/2212337095 >> 172.21.15.83:0/2212337095 conn(0x7f7424005ed0 msgr2=0x7f74240062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.458+0000 7f622f8b6640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.458+0000 7f622f8b6640 10 monclient: monmap: 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.472 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.458+0000 7f622f8b6640 1 librados: starting msgr at 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.458+0000 7f622f8b6640 1 librados: starting objecter 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.458+0000 7f622f8b6640 1 Processor -- start 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 1 -- start start 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 1 librados: setting wanted keys 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 1 librados: calling monclient init 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 10 monclient: init 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6228071c60 0x7f6228144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 10 monclient(hunting): picked mon.noname-c con 0x7f6228071c60 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 0x7f6228135b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 10 monclient(hunting): picked mon.noname-b con 0x7f622813e6b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.473 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.459+0000 7f622f8b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6228141890 0x7f6228136080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622de53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6228141890 0x7f6228136080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622de53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6228141890 0x7f6228136080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56016/0 (socket says 172.21.15.83:56016) 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 10 monclient(hunting): picked mon.noname-a con 0x7f6228141890 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622de53640 1 -- 172.21.15.83:0/252829616 learned_addr learned my addr 172.21.15.83:0/252829616 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6228072040 con 0x7f6228141890 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f622813eb20 con 0x7f622813e6b0 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f62280721c0 con 0x7f6228071c60 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622f8b6640 10 monclient(hunting): authenticate will time out at 1355.652596s 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622de53640 10 monclient(hunting): get_auth_request con 0x7f6228141890 auth_method 0 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622de53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622de53640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.462+0000 7f622de53640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.464+0000 7f622de53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.464+0000 7f622de53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.464+0000 7f622de53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.464+0000 7fd91dd6b640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.464+0000 7fd91dd6b640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.464+0000 7fd91dd6b640 10 monclient: monmap: 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.466158+0000 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.466158+0000 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.476 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.465+0000 7f7432dcf640 1 -- 172.21.15.83:0/2212337095 shutdown_connections 2024-03-25T02:00:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.465+0000 7f7432dcf640 1 -- 172.21.15.83:0/2212337095 wait complete. 2024-03-25T02:00:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.465+0000 7fd91dd6b640 1 Processor -- start 2024-03-25T02:00:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.465+0000 7fd91dd6b640 1 -- start start 2024-03-25T02:00:36.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.465+0000 7fd91dd6b640 10 monclient: init 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7f7432dcf640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7f7432dcf640 10 monclient: monmap: 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.478 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7f7432dcf640 1 librados: starting msgr at 2024-03-25T02:00:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7f7432dcf640 1 librados: starting objecter 2024-03-25T02:00:36.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7f7432dcf640 1 Processor -- start 2024-03-25T02:00:36.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-03-25T02:00:36.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9180b04a0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd9180b2df0 0x7fd918140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient(hunting): picked mon.noname-c con 0x7fd9180b2df0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fd918147350 0x7fd918140940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient(hunting): picked mon.noname-a con 0x7fd918147350 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9180b2410 con 0x7fd9180b04a0 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fd918140e80 con 0x7fd918147350 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fd9180b26f0 con 0x7fd9180b2df0 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.466+0000 7fd91dd6b640 10 monclient(hunting): authenticate will time out at 1355.656860s 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 -- start start 2024-03-25T02:00:36.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 librados: setting wanted keys 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 librados: calling monclient init 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient: init 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f742402fc90 0x7f74240c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f742402fc90 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f7424032150 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7424034c10 0x7f74240b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f7424034c10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f74240319e0 con 0x7f742402fc90 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f74240cc7b0 con 0x7f7424032150 2024-03-25T02:00:36.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f74240d68f0 con 0x7f7424034c10 2024-03-25T02:00:36.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.467+0000 7f7432dcf640 10 monclient(hunting): authenticate will time out at 1355.657886s 2024-03-25T02:00:36.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.468+0000 7f622ce51640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 0x7f6228135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.468+0000 7fd9177fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd9180b04a0 0x7fd918146c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33332/0 (socket says 172.21.15.83:33332) 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.468+0000 7fd9177fe640 1 -- 172.21.15.83:0/835303222 learned_addr learned my addr 172.21.15.83:0/835303222 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7fd916ffd640 1 --2- 172.21.15.83:0/835303222 >> v2:172.21.15.153:3300/0 conn(0x7fd9180b2df0 0x7fd918140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f74315cc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 1 -- 172.21.15.83:0/252829616 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6228071c60 msgr2=0x7f6228144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6228071c60 0x7f6228144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 1 -- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 msgr2=0x7f6228135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 0x7f6228135b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 1 monclient: found mon.noname-a 2024-03-25T02:00:36.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.469+0000 7f622de53640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6228136f30 con 0x7f6228141890 2024-03-25T02:00:36.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.470+0000 7fd916ffd640 10 monclient(hunting): get_auth_request con 0x7fd9180b2df0 auth_method 0 2024-03-25T02:00:36.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.470+0000 7fd916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.470+0000 7fd916ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.470+0000 7fd916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7f74315cc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240b0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40638/0 (socket says 172.21.15.83:40638) 2024-03-25T02:00:36.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7f74315cc640 1 -- 172.21.15.83:0/2587036759 learned_addr learned my addr 172.21.15.83:0/2587036759 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7f74315cc640 10 monclient(hunting): get_auth_request con 0x7f7424032150 auth_method 0 2024-03-25T02:00:36.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7f74315cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7f74315cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7f74315cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7fd9167fc640 1 -- 172.21.15.83:0/835303222 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105995196 0 0) 0x7fd9180b2410 con 0x7fd9180b04a0 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7fd9167fc640 10 monclient(hunting): _negotiate 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.471+0000 7fd9167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd9167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd9167fc640 10 monclient(hunting): my global_id is 4357 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd9167fc640 1 -- 172.21.15.83:0/835303222 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd900003520 con 0x7fd9180b04a0 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd917fff640 1 --2- 172.21.15.83:0/835303222 >> v2:172.21.15.83:3300/0 conn(0x7fd918147350 0x7fd918140940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f7431dcd640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f742402fc90 0x7f74240c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f7431dcd640 10 monclient(hunting): get_auth_request con 0x7f742402fc90 auth_method 0 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f7431dcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f7431dcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f7431dcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd916ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd916ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd916ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd917fff640 10 monclient(hunting): get_auth_request con 0x7fd918147350 auth_method 0 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd917fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7fd917fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f74315cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f74315cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.490 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.472+0000 7f74315cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.473+0000 7fd917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.473+0000 7fd917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.473+0000 7fd917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 1 -- 172.21.15.83:0/835303222 >> v2:172.21.15.153:3300/0 conn(0x7fd9180b2df0 msgr2=0x7fd918140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 1 --2- 172.21.15.83:0/835303222 >> v2:172.21.15.153:3300/0 conn(0x7fd9180b2df0 0x7fd918140400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 1 -- 172.21.15.83:0/835303222 >> v1:172.21.15.153:6789/0 conn(0x7fd9180b04a0 legacy=0x7fd918146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 1 monclient: found mon.noname-a 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.491 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.475+0000 7fd917fff640 1 -- 172.21.15.83:0/835303222 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9180b29d0 con 0x7fd918147350 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 1 -- 172.21.15.83:0/2587036759 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7424034c10 msgr2=0x7f74240b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7424034c10 0x7f74240b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 1 -- 172.21.15.83:0/2587036759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f742402fc90 msgr2=0x7f74240c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f742402fc90 0x7f74240c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.492 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.477+0000 7f74315cc640 1 -- 172.21.15.83:0/2587036759 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742402aaa0 con 0x7f7424032150 2024-03-25T02:00:36.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.481+0000 7f622de53640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6228141890 0x7f6228136080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f622000c9f0 tx=0x7f622000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.481+0000 7f621dffb640 10 client.?.objecter ms_handle_connect 0x7f6228141890 2024-03-25T02:00:36.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.481+0000 7f621dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.493 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.483+0000 7fd917fff640 1 --2- 172.21.15.83:0/835303222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd918147350 0x7fd918140940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd910009d90 tx=0x7fd91002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b809677202cbe09 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.486+0000 7f74315cc640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240b0220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f742800c430 tx=0x7f742800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.486+0000 7f74227fc640 10 client.?.objecter ms_handle_connect 0x7f7424032150 2024-03-25T02:00:36.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.486+0000 7f74227fc640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.487+0000 7f927359e640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.487+0000 7f927359e640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.487+0000 7f927359e640 10 monclient: monmap: 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.489316+0000 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.489316+0000 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.495 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f74227fc640 1 -- 172.21.15.83:0/2587036759 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428017070 con 0x7f7424032150 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f74227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f74227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f74227fc640 10 monclient: dump: 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f74227fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f74227fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f74227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 5 monclient: authenticate success, global_id 4363 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 10 monclient: _renew_subs 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742402ad10 con 0x7f7424032150 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 10 monclient: _renew_subs 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74240b3660 con 0x7f7424032150 2024-03-25T02:00:36.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f7432dcf640 1 librados: init done 2024-03-25T02:00:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f927359e640 1 Processor -- start 2024-03-25T02:00:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f927359e640 1 -- start start 2024-03-25T02:00:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.488+0000 7f927359e640 10 monclient: init 2024-03-25T02:00:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.489+0000 7f7432dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.489+0000 7f7432dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.489+0000 7f7432dcf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.489+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74240c6690 con 0x7f7424032150 2024-03-25T02:00:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.489+0000 7f927359e640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.489+0000 7f927359e640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-25T02:00:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92740a9ab0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9274140650 0x7f9274143090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 10 monclient(hunting): picked mon.noname-c con 0x7f9274140650 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740aacf0 0x7f92741435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f29167fc640 1 -- 172.21.15.83:0/367446305 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f291803e850 con 0x7f29200b04a0 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 10 monclient(hunting): picked mon.noname-b con 0x7f92740aacf0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92740abd70 con 0x7f92740a9ab0 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f92740a9e90 con 0x7f9274140650 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f92740aa140 con 0x7f92740aacf0 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f927359e640 10 monclient(hunting): authenticate will time out at 1355.680187s 2024-03-25T02:00:36.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f74227fc640 1 -- 172.21.15.83:0/2587036759 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7428004a80 con 0x7f7424032150 2024-03-25T02:00:36.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f29167fc640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f290406da70 0x7f290406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f74227fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f74227fc640 1 -- 172.21.15.83:0/2587036759 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428021060 con 0x7f7424032150 2024-03-25T02:00:36.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f74227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f292534a640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f290406da70 0x7f290406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.490+0000 7f292534a640 10 monclient: get_auth_request con 0x7f290406da70 auth_method 0 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.491+0000 7f7423fff640 4 set_mon_vals no callback set 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.491+0000 7f74227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.491+0000 7f74227fc640 10 monclient: dump: 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.502 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.491+0000 7f292534a640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f290406da70 0x7f290406ff30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29200b6d90 tx=0x7f291c009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.503 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.491+0000 7f29167fc640 10 client.4271.objecter ms_handle_connect 0x7f290406da70 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.491+0000 7f621dffb640 1 -- 172.21.15.83:0/252829616 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220017070 con 0x7f6228141890 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: dump: 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.505 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 1 -- 172.21.15.83:0/252829616 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f622000ab70 con 0x7f6228141890 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 1 -- 172.21.15.83:0/252829616 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220012400 con 0x7f6228141890 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621f7fe640 4 set_mon_vals no callback set 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f621dffb640 10 monclient: dump: 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.506 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 5 monclient: authenticate success, global_id 4335 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 10 monclient: _renew_subs 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62281365c0 con 0x7f6228141890 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 10 monclient: _renew_subs 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.507 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62281374f0 con 0x7f6228141890 2024-03-25T02:00:36.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.492+0000 7f622f8b6640 1 librados: init done 2024-03-25T02:00:36.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.493+0000 7f622f8b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.493+0000 7f622f8b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.493+0000 7f622f8b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.493+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6228075810 con 0x7f6228141890 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 1 -- 172.21.15.83:0/835303222 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910039070 con 0x7fd918147350 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 10 monclient: dump: 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.510 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 1 -- 172.21.15.83:0/835303222 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91002cc70 con 0x7fd918147350 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 1 -- 172.21.15.83:0/835303222 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910034660 con 0x7fd918147350 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.494+0000 7fd9167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd9167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd9167fc640 10 monclient: dump: 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.511 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 5 monclient: authenticate success, global_id 4338 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 4 set_mon_vals no callback set 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 10 monclient: shutdown 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 1 -- 172.21.15.83:0/835303222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd918147350 msgr2=0x7fd918140940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/835303222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd918147350 0x7fd918140940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd910009d90 tx=0x7fd91002bec0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 1 -- 172.21.15.83:0/835303222 shutdown_connections 2024-03-25T02:00:36.512 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/835303222 >> v2:172.21.15.153:3300/0 conn(0x7fd9180b2df0 0x7fd918140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/835303222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd918147350 0x7fd918140940 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.495+0000 7fd91dd6b640 1 -- 172.21.15.83:0/835303222 >> 172.21.15.83:0/835303222 conn(0x7fd9180b8a00 msgr2=0x7fd9180b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.496+0000 7fd91dd6b640 1 -- 172.21.15.83:0/835303222 shutdown_connections 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.496+0000 7fd91dd6b640 1 -- 172.21.15.83:0/835303222 wait complete. 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.496+0000 7fd91dd6b640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.496+0000 7fd91dd6b640 10 monclient: monmap: 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.513 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.496+0000 7fd91dd6b640 1 librados: starting msgr at 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.496+0000 7fd91dd6b640 1 librados: starting objecter 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.497+0000 7fd91dd6b640 1 Processor -- start 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.497+0000 7fd91dd6b640 1 -- start start 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.497+0000 7fd91dd6b640 1 librados: setting wanted keys 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.497+0000 7fd91dd6b640 1 librados: calling monclient init 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.497+0000 7fd91dd6b640 10 monclient: init 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9180b04a0 0x7fd9181427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): picked mon.noname-a con 0x7fd9180b04a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd9180b2df0 0x7fd918071180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): picked mon.noname-c con 0x7fd9180b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd918147350 0x7fd91806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): picked mon.noname-b con 0x7fd918147350 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd9180b0a00 con 0x7fd9180b04a0 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9180b0880 con 0x7fd918147350 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fd91814daf0 con 0x7fd9180b2df0 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.515 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.498+0000 7fd91dd6b640 10 monclient(hunting): authenticate will time out at 1355.688225s 2024-03-25T02:00:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.506+0000 7f6a2a96b640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.506+0000 7f6a2a96b640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.506+0000 7f6a2a96b640 10 monclient: monmap: 2024-03-25T02:00:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.508000+0000 2024-03-25T02:00:36.517 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.508000+0000 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.507+0000 7f6a2a96b640 1 Processor -- start 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.507+0000 7f6a2a96b640 1 -- start start 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.507+0000 7f6a2a96b640 10 monclient: init 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-03-25T02:00:36.518 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a240b04a0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6a240b2df0 0x7f6a24140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient(hunting): picked mon.noname-a con 0x7f6a240b2df0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6a24147350 0x7f6a24140940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient(hunting): picked mon.noname-c con 0x7f6a24147350 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a240b2410 con 0x7f6a240b04a0 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6a24140e80 con 0x7f6a240b2df0 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f6a240b26f0 con 0x7f6a24147350 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.508+0000 7f6a2a96b640 10 monclient(hunting): authenticate will time out at 1355.698587s 2024-03-25T02:00:36.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.509+0000 7f74227fc640 1 -- 172.21.15.83:0/2587036759 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f742801a070 con 0x7f7424032150 2024-03-25T02:00:36.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.509+0000 7f74227fc640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f741006da50 0x7f741006ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.510+0000 7f7431dcd640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f741006da50 0x7f741006ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.510+0000 7f7431dcd640 10 monclient: get_auth_request con 0x7f741006da50 auth_method 0 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.510+0000 7f74227fc640 10 client.4363.objecter ms_handle_connect 0x7f741006da50 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.510+0000 7f7431dcd640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f741006da50 0x7f741006ff10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f742c0506f0 tx=0x7f742c06f040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.511+0000 7f74227fc640 1 -- 172.21.15.83:0/2587036759 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f7428056b20 con 0x7f7424032150 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.511+0000 7f74227fc640 10 client.4363.objecter ms_dispatch 0x7f742c000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.511+0000 7f74227fc640 3 client.4363.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.511+0000 7f74227fc640 3 client.4363.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.511+0000 7f74227fc640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.536+0000 7f621dffb640 1 -- 172.21.15.83:0/252829616 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f622001a070 con 0x7f6228141890 2024-03-25T02:00:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.536+0000 7f621dffb640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f620c06da70 0x7f620c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.537+0000 7f622d652640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f620c06da70 0x7f620c06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.537+0000 7f622d652640 10 monclient: get_auth_request con 0x7f620c06da70 auth_method 0 2024-03-25T02:00:36.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.537+0000 7f622d652640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f620c06da70 0x7f620c06ff30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6218000f60 tx=0x7f62180079b0 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.537+0000 7f621dffb640 10 client.4335.objecter ms_handle_connect 0x7f620c06da70 2024-03-25T02:00:36.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.545+0000 7f370c960640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.545+0000 7f370c960640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.545+0000 7f370c960640 10 monclient: monmap: 2024-03-25T02:00:36.547 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.547 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.547 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.546930+0000 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.546930+0000 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f370c960640 1 Processor -- start 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f70a7a45640 10 monclient: get_monmap_and_config 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f70a7a45640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f70a7a45640 10 monclient: monmap: 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:36.547575+0000 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:36.547575+0000 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.549 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f370c960640 1 -- start start 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f370c960640 10 monclient: init 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f70a7a45640 1 Processor -- start 2024-03-25T02:00:36.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f70a7a45640 1 -- start start 2024-03-25T02:00:36.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.546+0000 7f70a7a45640 10 monclient: init 2024-03-25T02:00:36.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-03-25T02:00:36.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f37080b04a0 0x7f3708146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient(hunting): picked mon.noname-c con 0x7f37080b04a0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient(hunting): picked mon.noname-b con 0x7f37080b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3708140400 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37080b2410 con 0x7f3708140400 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f37080b26f0 con 0x7f37080b04a0 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f37080b29d0 con 0x7f37080b2df0 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f370c960640 10 monclient(hunting): authenticate will time out at 1355.737171s 2024-03-25T02:00:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f70a00b04a0 0x7f70a0146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient(hunting): picked mon.noname-a con 0x7f70a00b04a0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f70a00b2df0 0x7f70a0147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient(hunting): picked mon.noname-c con 0x7f70a00b2df0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70a0140400 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a00b2410 con 0x7f70a0140400 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f70a00b26f0 con 0x7f70a00b04a0 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f70a00b29d0 con 0x7f70a00b2df0 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.547+0000 7f70a7a45640 10 monclient(hunting): authenticate will time out at 1355.737591s 2024-03-25T02:00:36.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.581+0000 7f621dffb640 1 -- 172.21.15.83:0/252829616 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f62200573d0 con 0x7f6228141890 2024-03-25T02:00:36.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.581+0000 7f621dffb640 10 client.4335.objecter ms_dispatch 0x7f6228000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.581+0000 7f621dffb640 3 client.4335.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.583 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.581+0000 7f621dffb640 3 client.4335.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.581+0000 7f621dffb640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.581+0000 7fd9177fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9180b04a0 0x7fd9181427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.584 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.581+0000 7f6a237fe640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6a240b2df0 0x7f6a24140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.744+0000 7f9271d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9274140650 0x7f9274143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7fd917fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd918147350 0x7fd91806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7fd917fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd918147350 0x7fd91806eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40658/0 (socket says 172.21.15.83:40658) 2024-03-25T02:00:36.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7fd917fff640 1 -- 172.21.15.83:0/833486044 learned_addr learned my addr 172.21.15.83:0/833486044 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7fd917fff640 10 monclient(hunting): get_auth_request con 0x7fd918147350 auth_method 0 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7fd917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7fd917fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7fd917fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7f9271d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9274140650 0x7f9274143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40650/0 (socket says 172.21.15.83:40650) 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7f9271d9b640 1 -- 172.21.15.83:0/1569524394 learned_addr learned my addr 172.21.15.83:0/1569524394 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7f9271d9b640 10 monclient(hunting): get_auth_request con 0x7f9274140650 auth_method 0 2024-03-25T02:00:36.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7f9271d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7f9271d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.745+0000 7f9271d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.746+0000 7f9271d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.746+0000 7f9271d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.746+0000 7f9271d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2024-03-25T02:00:36.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 1 -- 172.21.15.83:0/1569524394 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740aacf0 msgr2=0x7f92741435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 1 --2- 172.21.15.83:0/1569524394 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740aacf0 0x7f92741435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 1 -- 172.21.15.83:0/1569524394 >> v1:172.21.15.153:6789/0 conn(0x7f92740a9ab0 legacy=0x7f927413ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.750 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.748+0000 7f9271d9b640 1 -- 172.21.15.83:0/1569524394 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92740ac050 con 0x7f9274140650 2024-03-25T02:00:36.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.749+0000 7fd917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.749+0000 7fd917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.749+0000 7fd917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.750+0000 7f370659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f37080b04a0 0x7f3708146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 1 -- 172.21.15.83:0/833486044 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd9180b2df0 msgr2=0x7fd918071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd9180b2df0 0x7fd918071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 1 -- 172.21.15.83:0/833486044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9180b04a0 msgr2=0x7fd9181427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9180b04a0 0x7fd9181427c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.752+0000 7fd917fff640 1 -- 172.21.15.83:0/833486044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd918070530 con 0x7fd918147350 2024-03-25T02:00:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.754+0000 7f370659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f37080b04a0 0x7f3708146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40684/0 (socket says 172.21.15.83:40684) 2024-03-25T02:00:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.754+0000 7f370659c640 1 -- 172.21.15.83:0/670634680 learned_addr learned my addr 172.21.15.83:0/670634680 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.754+0000 7f370659c640 10 monclient(hunting): get_auth_request con 0x7f37080b04a0 auth_method 0 2024-03-25T02:00:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.754+0000 7f370659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.754+0000 7f370659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.756 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.754+0000 7f370659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 1 -- 172.21.15.83:0/367446305 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f291807a9b0 con 0x7f29200b04a0 2024-03-25T02:00:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 10 client.4271.objecter ms_dispatch 0x7f2920000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 3 client.4271.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 3 client.4271.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 1 -- 172.21.15.83:0/367446305 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f29200b7bb0 con 0x7f29200b04a0 2024-03-25T02:00:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.760+0000 7f29167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.765+0000 7f370659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.765+0000 7f370659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.765+0000 7f370659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.766+0000 7f74227fc640 1 -- 172.21.15.83:0/2587036759 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f74240c6690 con 0x7f7424032150 2024-03-25T02:00:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.766+0000 7f74227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.766+0000 7f74227fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.766+0000 7f6a28f08640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6a24147350 0x7f6a24140940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-03-25T02:00:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 1 -- 172.21.15.83:0/670634680 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 msgr2=0x7f3708147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 1 --2- 172.21.15.83:0/670634680 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 1 -- 172.21.15.83:0/670634680 >> v1:172.21.15.83:6789/0 conn(0x7f3708140400 legacy=0x7f3708140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.767+0000 7f370659c640 1 -- 172.21.15.83:0/670634680 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37080b3380 con 0x7f37080b04a0 2024-03-25T02:00:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.768+0000 7f6a23fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6a240b04a0 0x7f6a24146c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33344/0 (socket says 172.21.15.83:33344) 2024-03-25T02:00:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.768+0000 7f6a23fff640 1 -- 172.21.15.83:0/617000329 learned_addr learned my addr 172.21.15.83:0/617000329 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.768+0000 7f70a4fe0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f70a00b2df0 0x7f70a0147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.769+0000 7f6a28f08640 10 monclient(hunting): get_auth_request con 0x7f6a24147350 auth_method 0 2024-03-25T02:00:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.769+0000 7f6a28f08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.769+0000 7f6a28f08640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.769+0000 7f6a28f08640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.769+0000 7f6a28f08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.769+0000 7f6a28f08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.769+0000 7f6a28f08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2024-03-25T02:00:36.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 1 -- 172.21.15.83:0/617000329 >> v2:172.21.15.83:3300/0 conn(0x7f6a240b2df0 msgr2=0x7f6a24140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 1 --2- 172.21.15.83:0/617000329 >> v2:172.21.15.83:3300/0 conn(0x7f6a240b2df0 0x7f6a24140400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 1 -- 172.21.15.83:0/617000329 >> v1:172.21.15.153:6789/0 conn(0x7f6a240b04a0 legacy=0x7f6a24146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.771+0000 7f6a28f08640 1 -- 172.21.15.83:0/617000329 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a240b29d0 con 0x7f6a24147350 2024-03-25T02:00:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.774+0000 7f9271d9b640 1 --2- 172.21.15.83:0/1569524394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9274140650 0x7f9274143090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9264009870 tx=0x7f926402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6da657dfd1388c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.778+0000 7f70a4fe0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f70a00b2df0 0x7f70a0147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40696/0 (socket says 172.21.15.83:40696) 2024-03-25T02:00:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.778+0000 7f70a4fe0640 1 -- 172.21.15.83:0/3940127456 learned_addr learned my addr 172.21.15.83:0/3940127456 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.778+0000 7f70a4fe0640 10 monclient(hunting): get_auth_request con 0x7f70a00b2df0 auth_method 0 2024-03-25T02:00:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.778+0000 7f70a4fe0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.778+0000 7f70a4fe0640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.778+0000 7f70a4fe0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.778+0000 7f370659c640 1 --2- 172.21.15.83:0/670634680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 0x7f3708146c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009870 tx=0x7f36f802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5b29d114c03d111 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 1 -- 172.21.15.83:0/1569524394 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92640317c0 con 0x7f9274140650 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: dump: 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.785 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 1 -- 172.21.15.83:0/1569524394 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9264037070 con 0x7f9274140650 2024-03-25T02:00:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927359e640 5 monclient: authenticate success, global_id 4366 2024-03-25T02:00:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 1 -- 172.21.15.83:0/1569524394 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264031b80 con 0x7f9274140650 2024-03-25T02:00:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927159a640 10 monclient: dump: 2024-03-25T02:00:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.788 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.788 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.788 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.788 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.788 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.788 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.789 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927359e640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f927359e640 4 set_mon_vals no callback set 2024-03-25T02:00:36.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f70a4fe0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f70a4fe0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.782+0000 7f70a4fe0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 10 monclient: shutdown 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 -- 172.21.15.83:0/1569524394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9274140650 msgr2=0x7f9274143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 --2- 172.21.15.83:0/1569524394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9274140650 0x7f9274143090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9264009870 tx=0x7f926402b860 comp rx=0 tx=0).stop 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 -- 172.21.15.83:0/1569524394 shutdown_connections 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 --2- 172.21.15.83:0/1569524394 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740aacf0 0x7f92741435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 --2- 172.21.15.83:0/1569524394 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9274140650 0x7f9274143090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 -- 172.21.15.83:0/1569524394 >> 172.21.15.83:0/1569524394 conn(0x7f92740b89f0 msgr2=0x7f92740b8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 -- 172.21.15.83:0/1569524394 shutdown_connections 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.783+0000 7f927359e640 1 -- 172.21.15.83:0/1569524394 wait complete. 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f927359e640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f927359e640 10 monclient: monmap: 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.792 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.793 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 1 -- 172.21.15.83:0/670634680 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8034070 con 0x7f37080b04a0 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: dump: 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.794 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 1 -- 172.21.15.83:0/670634680 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f8030840 con 0x7f37080b04a0 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370c960640 5 monclient: authenticate success, global_id 4372 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370c960640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370c960640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 1 -- 172.21.15.83:0/670634680 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8030c10 con 0x7f37080b04a0 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370c960640 4 set_mon_vals no callback set 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f370559a640 10 monclient: dump: 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.796 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f927359e640 1 librados: starting msgr at 2024-03-25T02:00:36.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.784+0000 7f927359e640 1 librados: starting objecter 2024-03-25T02:00:36.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f927359e640 1 Processor -- start 2024-03-25T02:00:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f927359e640 1 -- start start 2024-03-25T02:00:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f927359e640 1 librados: setting wanted keys 2024-03-25T02:00:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f927359e640 1 librados: calling monclient init 2024-03-25T02:00:36.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f927359e640 10 monclient: init 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 10 monclient: shutdown 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 1 -- 172.21.15.83:0/670634680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 msgr2=0x7f3708146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 1 --2- 172.21.15.83:0/670634680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 0x7f3708146c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009870 tx=0x7f36f802b980 comp rx=0 tx=0).stop 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 1 -- 172.21.15.83:0/670634680 shutdown_connections 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 1 --2- 172.21.15.83:0/670634680 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 1 --2- 172.21.15.83:0/670634680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 0x7f3708146c40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 1 -- 172.21.15.83:0/670634680 >> 172.21.15.83:0/670634680 conn(0x7f37080b8a00 msgr2=0x7f37080b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.785+0000 7f370c960640 1 -- 172.21.15.83:0/670634680 shutdown_connections 2024-03-25T02:00:36.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f370c960640 1 -- 172.21.15.83:0/670634680 wait complete. 2024-03-25T02:00:36.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740a9ab0 0x7f9274146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): picked mon.noname-c con 0x7f92740a9ab0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92740aacf0 0x7f92741d8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): picked mon.noname-b con 0x7f92740aacf0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9274140650 0x7f92741d8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): picked mon.noname-a con 0x7f9274140650 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9274143c90 con 0x7f9274140650 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9274143b10 con 0x7f92740aacf0 2024-03-25T02:00:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f927414d7f0 con 0x7f92740a9ab0 2024-03-25T02:00:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.786+0000 7f927359e640 10 monclient(hunting): authenticate will time out at 1355.976461s 2024-03-25T02:00:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.804 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 10 monclient: monmap: 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 1 librados: starting msgr at 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 1 librados: starting objecter 2024-03-25T02:00:36.805 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 1 Processor -- start 2024-03-25T02:00:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2024-03-25T02:00:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 1 -- 172.21.15.83:0/3940127456 >> v2:172.21.15.83:3300/0 conn(0x7f70a00b04a0 msgr2=0x7f70a0146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 1 --2- 172.21.15.83:0/3940127456 >> v2:172.21.15.83:3300/0 conn(0x7f70a00b04a0 0x7f70a0146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 1 -- 172.21.15.83:0/3940127456 >> v1:172.21.15.153:6789/0 conn(0x7f70a0140400 legacy=0x7f70a0140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.806 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f70a4fe0640 1 -- 172.21.15.83:0/3940127456 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a00b3380 con 0x7f70a00b2df0 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 1 -- start start 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 1 librados: setting wanted keys 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 1 librados: calling monclient init 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.787+0000 7f370c960640 10 monclient: init 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 0x7f3708143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): picked mon.noname-b con 0x7f37080b04a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): picked mon.noname-c con 0x7f37080b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3708140400 0x7f3708137300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): picked mon.noname-a con 0x7f3708140400 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f370814dc70 con 0x7f3708140400 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f370814daf0 con 0x7f37080b04a0 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f370814ddf0 con 0x7f37080b2df0 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.788+0000 7f370c960640 10 monclient(hunting): authenticate will time out at 1355.978352s 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.790+0000 7f6a28f08640 1 --2- 172.21.15.83:0/617000329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a24147350 0x7f6a24140940 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009d90 tx=0x7f6a1c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace01e9f2876f57a server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.794+0000 7f621dffb640 1 -- 172.21.15.83:0/252829616 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6228136f30 con 0x7f6228141890 2024-03-25T02:00:36.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.794+0000 7f621dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.794+0000 7f621dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:36.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 1 -- 172.21.15.83:0/617000329 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c039070 con 0x7f6a24147350 2024-03-25T02:00:36.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: dump: 2024-03-25T02:00:36.810 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.810 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 1 -- 172.21.15.83:0/617000329 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a1c02cc60 con 0x7f6a24147350 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 1 -- 172.21.15.83:0/617000329 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c034660 con 0x7f6a24147350 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a22ffd640 10 monclient: dump: 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.812 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.813 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f927259c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740a9ab0 0x7f9274146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f9271d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92740aacf0 0x7f92741d8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a2a96b640 5 monclient: authenticate success, global_id 4375 2024-03-25T02:00:36.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a2a96b640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.795+0000 7f6a2a96b640 4 set_mon_vals no callback set 2024-03-25T02:00:36.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f9271d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92740aacf0 0x7f92741d8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40700/0 (socket says 172.21.15.83:40700) 2024-03-25T02:00:36.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f9271d9b640 1 -- 172.21.15.83:0/2334861151 learned_addr learned my addr 172.21.15.83:0/2334861151 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f9271d9b640 10 monclient(hunting): get_auth_request con 0x7f92740aacf0 auth_method 0 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f9271d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f9271d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f9271d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 10 monclient: shutdown 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 1 -- 172.21.15.83:0/617000329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a24147350 msgr2=0x7f6a24140940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/617000329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a24147350 0x7f6a24140940 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009d90 tx=0x7f6a1c02beb0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 1 -- 172.21.15.83:0/617000329 shutdown_connections 2024-03-25T02:00:36.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/617000329 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a24147350 0x7f6a24140940 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/617000329 >> v2:172.21.15.83:3300/0 conn(0x7f6a240b2df0 0x7f6a24140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.796+0000 7f6a2a96b640 1 -- 172.21.15.83:0/617000329 >> 172.21.15.83:0/617000329 conn(0x7f6a240b8a00 msgr2=0x7f6a240b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7fd917fff640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd918147350 0x7fd91806eb90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd91002c4d0 tx=0x7fd910004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7fd8fffff640 10 client.?.objecter ms_handle_connect 0x7fd918147350 2024-03-25T02:00:36.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7fd8fffff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7f6a2a96b640 1 -- 172.21.15.83:0/617000329 shutdown_connections 2024-03-25T02:00:36.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7f6a2a96b640 1 -- 172.21.15.83:0/617000329 wait complete. 2024-03-25T02:00:36.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7f9271d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7f9271d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.817 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.797+0000 7f9271d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.798+0000 7f6a2a96b640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.798+0000 7f6a2a96b640 10 monclient: monmap: 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.818 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.798+0000 7f6a2a96b640 1 librados: starting msgr at 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.798+0000 7f6a2a96b640 1 librados: starting objecter 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.798+0000 7f3705d9b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.798+0000 7f6a2a96b640 1 Processor -- start 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 1 -- start start 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 1 librados: setting wanted keys 2024-03-25T02:00:36.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 1 librados: calling monclient init 2024-03-25T02:00:36.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 10 monclient: init 2024-03-25T02:00:36.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f927259c640 10 monclient(hunting): get_auth_request con 0x7f92740a9ab0 auth_method 0 2024-03-25T02:00:36.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f927259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f927259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f927259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f70a4fe0640 1 --2- 172.21.15.83:0/3940127456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 0x7f70a0147180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7090009870 tx=0x7f709002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e0b9ada1f99f9ba server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a240b04a0 0x7f6a24141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.799+0000 7f6a2a96b640 10 monclient(hunting): picked mon.noname-b con 0x7f6a240b04a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6a240b2df0 0x7f6a2413aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 10 monclient(hunting): picked mon.noname-c con 0x7f6a240b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a24147350 0x7f6a2413b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 10 monclient(hunting): picked mon.noname-a con 0x7f6a24147350 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6a240b0a00 con 0x7f6a24147350 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a240b0880 con 0x7f6a240b04a0 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2414daf0 con 0x7f6a240b2df0 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f6a2a96b640 10 monclient(hunting): authenticate will time out at 1355.990223s 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f927259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f927259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f927259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f3705d9b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708135c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54570/0 (socket says 172.21.15.83:54570) 2024-03-25T02:00:36.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f3705d9b640 1 -- 172.21.15.83:0/3652778816 learned_addr learned my addr 172.21.15.83:0/3652778816 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f3705d9b640 10 monclient(hunting): get_auth_request con 0x7f37080b2df0 auth_method 0 2024-03-25T02:00:36.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f3705d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f3705d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.800+0000 7f3705d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 1 -- 172.21.15.83:0/2334861151 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740a9ab0 msgr2=0x7f9274146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740a9ab0 0x7f9274146cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 1 -- 172.21.15.83:0/2334861151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9274140650 msgr2=0x7f92741d8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9274140650 0x7f92741d8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.802+0000 7f9271d9b640 1 -- 172.21.15.83:0/2334861151 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9274144a60 con 0x7f92740aacf0 2024-03-25T02:00:36.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.806+0000 7f3705d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.806+0000 7f3705d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.806+0000 7f3705d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.807+0000 7f3706d9d640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3708140400 0x7f3708137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.807+0000 7f6a23fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a240b04a0 0x7f6a24141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.808+0000 7f3706d9d640 10 monclient(hunting): get_auth_request con 0x7f3708140400 auth_method 0 2024-03-25T02:00:36.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.808+0000 7f3706d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.808+0000 7f3706d9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.808+0000 7f3706d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f6a28f08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a24147350 0x7f6a2413b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f6a23fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a240b04a0 0x7f6a24141d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40716/0 (socket says 172.21.15.83:40716) 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f6a23fff640 1 -- 172.21.15.83:0/3506585645 learned_addr learned my addr 172.21.15.83:0/3506585645 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f6a23fff640 10 monclient(hunting): get_auth_request con 0x7f6a240b04a0 auth_method 0 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f6a23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f6a23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f6a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f3706d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f3706d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.809+0000 7f3706d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.828 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 1 -- 172.21.15.83:0/833486044 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910044020 con 0x7fd918147350 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: dump: 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 1 -- 172.21.15.83:0/833486044 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd910035e50 con 0x7fd918147350 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 1 -- 172.21.15.83:0/833486044 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd910039040 con 0x7fd918147350 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd8fffff640 10 monclient: dump: 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.830 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 5 monclient: authenticate success, global_id 4369 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd915ffb640 4 set_mon_vals no callback set 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 10 monclient: _renew_subs 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd918070d60 con 0x7fd918147350 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 10 monclient: _renew_subs 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd918070a80 con 0x7fd918147350 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.810+0000 7fd91dd6b640 1 librados: init done 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f6a23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.832 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f6a23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f6a23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7fd91dd6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 1 -- 172.21.15.83:0/3940127456 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090034070 con 0x7f70a00b2df0 2024-03-25T02:00:36.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7fd91dd6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7fd91dd6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: dump: 2024-03-25T02:00:36.834 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.834 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 1 -- 172.21.15.83:0/3940127456 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7090030860 con 0x7f70a00b2df0 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.835 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 1 -- 172.21.15.83:0/3940127456 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090030c30 con 0x7f70a00b2df0 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f7097fff640 10 monclient: dump: 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f70a7a45640 5 monclient: authenticate success, global_id 4381 2024-03-25T02:00:36.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f70a7a45640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:36.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7f70a7a45640 4 set_mon_vals no callback set 2024-03-25T02:00:36.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.811+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd918075810 con 0x7fd918147350 2024-03-25T02:00:36.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 10 monclient: shutdown 2024-03-25T02:00:36.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 1 -- 172.21.15.83:0/3940127456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 msgr2=0x7f70a0147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3940127456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 0x7f70a0147180 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7090009870 tx=0x7f709002b9a0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 1 -- 172.21.15.83:0/3940127456 shutdown_connections 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3940127456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 0x7f70a0147180 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3940127456 >> v2:172.21.15.83:3300/0 conn(0x7f70a00b04a0 0x7f70a0146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f70a7a45640 1 -- 172.21.15.83:0/3940127456 >> 172.21.15.83:0/3940127456 conn(0x7f70a00b8a00 msgr2=0x7f70a00b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f9271d9b640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92740aacf0 0x7f92741d8910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f926402b8a0 tx=0x7f92640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f9262ffd640 10 client.?.objecter ms_handle_connect 0x7f92740aacf0 2024-03-25T02:00:36.838 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.812+0000 7f9262ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.813+0000 7f70a7a45640 1 -- 172.21.15.83:0/3940127456 shutdown_connections 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.813+0000 7f70a7a45640 1 -- 172.21.15.83:0/3940127456 wait complete. 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.813+0000 7f70a7a45640 10 monclient: build_initial_monmap 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 10 monclient: monmap: 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:36.839 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 1 librados: starting msgr at 2024-03-25T02:00:36.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 1 librados: starting objecter 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 1 Processor -- start 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 1 -- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 msgr2=0x7f3708143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 0x7f3708143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 1 -- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3708140400 msgr2=0x7f3708137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3708140400 0x7f3708137300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 1 monclient: found mon.noname-c 2024-03-25T02:00:36.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f3705d9b640 1 -- 172.21.15.83:0/3652778816 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37081399e0 con 0x7f37080b2df0 2024-03-25T02:00:36.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 1 -- start start 2024-03-25T02:00:36.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 1 librados: setting wanted keys 2024-03-25T02:00:36.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 1 librados: calling monclient init 2024-03-25T02:00:36.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.814+0000 7f70a7a45640 10 monclient: init 2024-03-25T02:00:36.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-03-25T02:00:36.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 1 -- 172.21.15.83:0/3506585645 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6a240b2df0 msgr2=0x7f6a2413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6a240b2df0 0x7f6a2413aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.843 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 1 -- 172.21.15.83:0/3506585645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a24147350 msgr2=0x7f6a2413b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a24147350 0x7f6a2413b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f70a7a45640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:36.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 1 monclient: found mon.noname-b 2024-03-25T02:00:36.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:36.844 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f6a23fff640 1 -- 172.21.15.83:0/3506585645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a24143b50 con 0x7f6a240b04a0 2024-03-25T02:00:36.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f70a7a45640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:00:36.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f70a7a45640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70a00b04a0 0x7f70a01437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f70a7a45640 10 monclient(hunting): picked mon.noname-a con 0x7f70a00b04a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:36.845 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f70a7a45640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 0x7f70a01318c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f70a7a45640 10 monclient(hunting): picked mon.noname-b con 0x7f70a00b2df0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.815+0000 7f70a7a45640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f70a0140400 0x7f70a01e58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 10 monclient(hunting): picked mon.noname-c con 0x7f70a0140400 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f70a014dc70 con 0x7f70a00b04a0 2024-03-25T02:00:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.846 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f70a014daf0 con 0x7f70a00b2df0 2024-03-25T02:00:36.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:36.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f70a014ddf0 con 0x7f70a0140400 2024-03-25T02:00:36.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:36.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.816+0000 7f70a7a45640 10 monclient(hunting): authenticate will time out at 1356.006041s 2024-03-25T02:00:36.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.817+0000 7f6a23fff640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a240b04a0 0x7f6a24141d20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400c400 tx=0x7f6a1400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.817+0000 7f6a20ff9640 10 client.?.objecter ms_handle_connect 0x7f6a240b04a0 2024-03-25T02:00:36.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.817+0000 7f6a20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.819+0000 7f3705d9b640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708135c70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00b4a0 tx=0x7f36fc00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.819+0000 7f36f6ffd640 10 client.?.objecter ms_handle_connect 0x7f37080b2df0 2024-03-25T02:00:36.850 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.819+0000 7f36f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.826+0000 7f70a5fe2640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f70a0140400 0x7f70a01e58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 1 -- 172.21.15.83:0/3652778816 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc014070 con 0x7f37080b2df0 2024-03-25T02:00:36.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:36.851 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 10 monclient: dump: 2024-03-25T02:00:36.851 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.851 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 1 -- 172.21.15.83:0/3652778816 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36fc010610 con 0x7f37080b2df0 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f370c960640 5 monclient: authenticate success, global_id 4277 2024-03-25T02:00:36.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f370c960640 10 monclient: _renew_subs 2024-03-25T02:00:36.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f370c960640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3708139c50 con 0x7f37080b2df0 2024-03-25T02:00:36.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f36f6ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f3704d99640 4 set_mon_vals no callback set 2024-03-25T02:00:36.853 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.827+0000 7f370c960640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f70a5fe2640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f70a0140400 0x7f70a01e58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54588/0 (socket says 172.21.15.83:54588) 2024-03-25T02:00:36.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f70a5fe2640 1 -- 172.21.15.83:0/3320647035 learned_addr learned my addr 172.21.15.83:0/3320647035 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:36.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f70a5fe2640 10 monclient(hunting): get_auth_request con 0x7f70a0140400 auth_method 0 2024-03-25T02:00:36.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f70a5fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f70a5fe2640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f36f6ffd640 1 -- 172.21.15.83:0/3652778816 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc010a00 con 0x7f37080b2df0 2024-03-25T02:00:36.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f36f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f36f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f36f6ffd640 10 monclient: dump: 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f370c960640 10 monclient: _renew_subs 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f370c960640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37081380a0 con 0x7f37080b2df0 2024-03-25T02:00:36.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f370c960640 1 librados: init done 2024-03-25T02:00:36.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.828+0000 7f70a5fe2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.829+0000 7f70a4fe0640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 0x7f70a01318c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.829+0000 7f370c960640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.829+0000 7f370c960640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.829+0000 7f370c960640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.829+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3708075810 con 0x7f37080b2df0 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.830+0000 7f70a4fe0640 10 monclient(hunting): get_auth_request con 0x7f70a00b2df0 auth_method 0 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.830+0000 7f70a4fe0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.830+0000 7f70a4fe0640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.830+0000 7f70a4fe0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.840+0000 7f70a57e1640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70a00b04a0 0x7f70a01437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.840+0000 7f70a57e1640 10 monclient(hunting): get_auth_request con 0x7f70a00b04a0 auth_method 0 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.840+0000 7f70a57e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.840+0000 7f70a57e1640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.840+0000 7f70a57e1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:36.861 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.841+0000 7f70a57e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.841+0000 7f70a57e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.841+0000 7f70a57e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 1 -- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f70a0140400 msgr2=0x7f70a01e58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f70a0140400 0x7f70a01e58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 1 -- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 msgr2=0x7f70a01318c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 0x7f70a01318c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 10 monclient: _finish_hunting 0 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 1 monclient: found mon.noname-a 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.844+0000 7f70a57e1640 1 -- 172.21.15.83:0/3320647035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a0132cb0 con 0x7f70a00b04a0 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.846+0000 7f70a57e1640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70a00b04a0 0x7f70a01437e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f709c00b420 tx=0x7f709c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.846+0000 7f7095ffb640 10 client.?.objecter ms_handle_connect 0x7f70a00b04a0 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.846+0000 7f7095ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.850+0000 7f7095ffb640 1 -- 172.21.15.83:0/3320647035 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c011830 con 0x7f70a00b04a0 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 10 monclient: dump: 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.863 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 1 -- 172.21.15.83:0/3320647035 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f709c016070 con 0x7f70a00b04a0 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f70a7a45640 5 monclient: authenticate success, global_id 4347 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f7095ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.851+0000 7f70977fe640 4 set_mon_vals no callback set 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f7095ffb640 1 -- 172.21.15.83:0/3320647035 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c011c10 con 0x7f70a00b04a0 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 10 monclient: _renew_subs 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a0132f90 con 0x7f70a00b04a0 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f7095ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f7095ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f7095ffb640 10 monclient: dump: 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:36.865 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 10 monclient: _renew_subs 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a0141000 con 0x7f70a00b04a0 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.852+0000 7f70a7a45640 1 librados: init done 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.853+0000 7f70a7a45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.853+0000 7f70a7a45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.853+0000 7f70a7a45640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:36.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.853+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a00b1e50 con 0x7f70a00b04a0 2024-03-25T02:00:36.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.867+0000 7f36f6ffd640 1 -- 172.21.15.83:0/3652778816 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f36fc017070 con 0x7f37080b2df0 2024-03-25T02:00:36.869 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.867+0000 7f36f6ffd640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f36e806da70 0x7f36e806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.867+0000 7f370659c640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f36e806da70 0x7f36e806ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.868+0000 7f370659c640 10 monclient: get_auth_request con 0x7f36e806da70 auth_method 0 2024-03-25T02:00:36.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.868+0000 7f370659c640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f36e806da70 0x7f36e806ff30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f36f8036000 tx=0x7f36f8030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.868+0000 7f36f6ffd640 10 client.4277.objecter ms_handle_connect 0x7f36e806da70 2024-03-25T02:00:36.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.873+0000 7f7095ffb640 1 -- 172.21.15.83:0/3320647035 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f709c025430 con 0x7f70a00b04a0 2024-03-25T02:00:36.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.873+0000 7f7095ffb640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f708406da20 0x7f708406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:36.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f70a4fe0640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f708406da20 0x7f708406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:36.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f70a4fe0640 10 monclient: get_auth_request con 0x7f708406da20 auth_method 0 2024-03-25T02:00:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f36f6ffd640 1 -- 172.21.15.83:0/3652778816 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f36fc0570c0 con 0x7f37080b2df0 2024-03-25T02:00:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f36f6ffd640 10 client.4277.objecter ms_dispatch 0x7f3708000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f36f6ffd640 3 client.4277.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f36f6ffd640 3 client.4277.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f7095ffb640 10 client.4347.objecter ms_handle_connect 0x7f708406da20 2024-03-25T02:00:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.874+0000 7f70a4fe0640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f708406da20 0x7f708406fee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7090036000 tx=0x7f709003b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:36.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.875+0000 7f36f6ffd640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.900+0000 7f7095ffb640 1 -- 172.21.15.83:0/3320647035 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f709c057a50 con 0x7f70a00b04a0 2024-03-25T02:00:36.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.900+0000 7f7095ffb640 10 client.4347.objecter ms_dispatch 0x7f70a0000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:36.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.900+0000 7f7095ffb640 3 client.4347.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:36.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.900+0000 7f7095ffb640 3 client.4347.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:36.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.900+0000 7f7095ffb640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T02:00:36.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.965+0000 7f2927daf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-25T02:00:36.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.965+0000 7f2927daf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-25T02:00:36.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.965+0000 7f2927daf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:36.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:36.965+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f2920075810 con 0x7f29200b04a0 2024-03-25T02:00:37.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.001+0000 7f622f8b6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-25T02:00:37.003 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.001+0000 7f622f8b6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-25T02:00:37.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.001+0000 7f622f8b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:37.004 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.001+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f622806d6b0 con 0x7f6228141890 2024-03-25T02:00:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.032+0000 7f7432dcf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-25T02:00:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.032+0000 7f7432dcf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-25T02:00:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.032+0000 7f7432dcf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.032+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f74240052b0 con 0x7f7424032150 2024-03-25T02:00:37.072 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.070+0000 7fd8fffff640 1 -- 172.21.15.83:0/833486044 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fd910047020 con 0x7fd918147350 2024-03-25T02:00:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.071+0000 7fd8fffff640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd90406da20 0x7fd90406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.071+0000 7fd9177fe640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd90406da20 0x7fd90406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:37.073 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.071+0000 7fd9177fe640 10 monclient: get_auth_request con 0x7fd90406da20 auth_method 0 2024-03-25T02:00:37.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.072+0000 7fd8fffff640 10 client.4369.objecter ms_handle_connect 0x7fd90406da20 2024-03-25T02:00:37.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.072+0000 7fd9177fe640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd90406da20 0x7fd90406fee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd9080011e0 tx=0x7fd908009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:37.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.076+0000 7fd8fffff640 1 -- 172.21.15.83:0/833486044 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fd91007a080 con 0x7fd918147350 2024-03-25T02:00:37.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.076+0000 7fd8fffff640 10 client.4369.objecter ms_dispatch 0x7fd918000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:37.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.076+0000 7fd8fffff640 3 client.4369.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:37.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.076+0000 7fd8fffff640 3 client.4369.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:37.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.077+0000 7fd8fffff640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 1 -- 172.21.15.83:0/3506585645 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14017070 con 0x7f6a240b04a0 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: dump: 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:37.090 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: _finish_auth 0 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: _check_auth_tickets 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 1 -- 172.21.15.83:0/3506585645 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a14005ce0 con 0x7f6a240b04a0 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 1 -- 172.21.15.83:0/3506585645 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a140072f0 con 0x7f6a240b04a0 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a20ff9640 10 monclient: dump: 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.091 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 5 monclient: authenticate success, global_id 4387 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 10 monclient: _renew_subs 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a227fc640 4 set_mon_vals no callback set 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a24143e30 con 0x7f6a240b04a0 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 10 monclient: _renew_subs 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a24142d80 con 0x7f6a240b04a0 2024-03-25T02:00:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7f6a2a96b640 1 librados: init done 2024-03-25T02:00:37.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7fd8fffff640 1 -- 172.21.15.83:0/833486044 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd91004e030 con 0x7fd918147350 2024-03-25T02:00:37.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7fd8fffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.087+0000 7fd8fffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:37.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.088+0000 7f6a2a96b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.088+0000 7f6a2a96b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.088+0000 7f6a2a96b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.093 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.088+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a24075810 con 0x7f6a240b04a0 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 1 -- 172.21.15.83:0/2334861151 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264032860 con 0x7f92740aacf0 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: dump: 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:37.095 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 1 -- 172.21.15.83:0/2334861151 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9264032a20 con 0x7f92740aacf0 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f927359e640 5 monclient: authenticate success, global_id 4384 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f927359e640 10 monclient: _renew_subs 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f927359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9274144260 con 0x7f92740aacf0 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 1 -- 172.21.15.83:0/2334861151 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264032e10 con 0x7f92740aacf0 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:37.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9262ffd640 10 monclient: dump: 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:37.097 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.089+0000 7f9270d99640 4 set_mon_vals no callback set 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.090+0000 7f927359e640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.090+0000 7f927359e640 10 monclient: _renew_subs 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.090+0000 7f927359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.090+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9274145300 con 0x7f92740aacf0 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.090+0000 7f927359e640 1 librados: init done 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.091+0000 7f927359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.091+0000 7f927359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.091+0000 7f927359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.091+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f927404f940 con 0x7f92740aacf0 2024-03-25T02:00:37.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.137+0000 7f36f6ffd640 1 -- 172.21.15.83:0/3652778816 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f36fc059020 con 0x7f37080b2df0 2024-03-25T02:00:37.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.137+0000 7f36f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.139 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.137+0000 7f36f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.150+0000 7f29167fc640 1 -- 172.21.15.83:0/367446305 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f291807d010 con 0x7f29200b04a0 2024-03-25T02:00:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.151+0000 7f29167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-25T02:00:37.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.151+0000 7f29167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.153 INFO:teuthology.orchestra.run.smithi083.stdout:85899345928 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 10 librados: watch_flush enter 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 10 librados: watch_flush exit 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 10 client.4271.objecter successfully canceled tick 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f290406da70 msgr2=0x7f290406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f290406da70 0x7f290406ff30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29200b6d90 tx=0x7f291c009240 comp rx=0 tx=0).stop 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 10 monclient: shutdown 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 msgr2=0x7f2920145d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920145d90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2918009e80 tx=0x7f2918004a40 comp rx=0 tx=0).stop 2024-03-25T02:00:37.156 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 shutdown_connections 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f290406da70 0x7f290406ff30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2920140400 0x7f2920070f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29200b2df0 0x7f292006ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 --2- 172.21.15.83:0/367446305 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f29200b04a0 0x7f2920145d90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 >> 172.21.15.83:0/367446305 conn(0x7f29200b8a00 msgr2=0x7f2920146b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.153+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 shutdown_connections 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.154+0000 7f2927daf640 1 -- 172.21.15.83:0/367446305 wait complete. 2024-03-25T02:00:37.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.154+0000 7f2927daf640 1 librados: shutdown 2024-03-25T02:00:37.170 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345928 for osd.2 2024-03-25T02:00:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.205+0000 7f7095ffb640 1 -- 172.21.15.83:0/3320647035 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f70a00b1e50 con 0x7f70a00b04a0 2024-03-25T02:00:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.205+0000 7f7095ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.205+0000 7f7095ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:37.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.222+0000 7f621dffb640 1 -- 172.21.15.83:0/252829616 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6220059330 con 0x7f6228141890 2024-03-25T02:00:37.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.222+0000 7f621dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-25T02:00:37.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.222+0000 7f621dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.224 INFO:teuthology.orchestra.run.smithi083.stdout:64424509445 2024-03-25T02:00:37.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.224+0000 7f622f8b6640 10 librados: watch_flush enter 2024-03-25T02:00:37.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 10 librados: watch_flush exit 2024-03-25T02:00:37.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 10 client.4335.objecter successfully canceled tick 2024-03-25T02:00:37.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f620c06da70 msgr2=0x7f620c06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f620c06da70 0x7f620c06ff30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6218000f60 tx=0x7f62180079b0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 10 monclient: shutdown 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6228141890 msgr2=0x7f6228136080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6228141890 0x7f6228136080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f622000c9f0 tx=0x7f622000ceb0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 shutdown_connections 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f620c06da70 0x7f620c06ff30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6228141890 0x7f6228136080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f622813e6b0 0x7f6228135b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 --2- 172.21.15.83:0/252829616 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6228071c60 0x7f6228144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 >> 172.21.15.83:0/252829616 conn(0x7f62280b69c0 msgr2=0x7f62280b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 shutdown_connections 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 -- 172.21.15.83:0/252829616 wait complete. 2024-03-25T02:00:37.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.225+0000 7f622f8b6640 1 librados: shutdown 2024-03-25T02:00:37.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.0 2024-03-25T02:00:37.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.280+0000 7fd91dd6b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-25T02:00:37.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.280+0000 7fd91dd6b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-25T02:00:37.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.280+0000 7fd91dd6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.280+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd918072b10 con 0x7fd918147350 2024-03-25T02:00:37.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.339+0000 7f6a20ff9640 1 -- 172.21.15.83:0/3506585645 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f6a1401a070 con 0x7f6a240b04a0 2024-03-25T02:00:37.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.339+0000 7f6a20ff9640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6a1006da70 0x7f6a1006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:37.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.339+0000 7f6a237fe640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6a1006da70 0x7f6a1006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:37.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.339+0000 7f6a237fe640 10 monclient: get_auth_request con 0x7f6a1006da70 auth_method 0 2024-03-25T02:00:37.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.340+0000 7f6a20ff9640 10 client.4387.objecter ms_handle_connect 0x7f6a1006da70 2024-03-25T02:00:37.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.340+0000 7f6a237fe640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6a1006da70 0x7f6a1006ff30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006fd0 tx=0x7f6a18008040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:37.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.340+0000 7f370c960640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:37.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.340+0000 7f370c960640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:37.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.340+0000 7f370c960640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:37.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.340+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f370814d8d0 con 0x7f37080b2df0 2024-03-25T02:00:37.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.341+0000 7f6a20ff9640 1 -- 172.21.15.83:0/3506585645 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f6a14056ba0 con 0x7f6a240b04a0 2024-03-25T02:00:37.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.341+0000 7f6a20ff9640 10 client.4387.objecter ms_dispatch 0x7f6a24000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:37.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.341+0000 7f6a20ff9640 3 client.4387.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:37.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.341+0000 7f6a20ff9640 3 client.4387.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:37.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.341+0000 7f6a20ff9640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-25T02:00:37.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.344+0000 7f36f6ffd640 1 -- 172.21.15.83:0/3652778816 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f36fc05d0c0 con 0x7f37080b2df0 2024-03-25T02:00:37.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.344+0000 7f36f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:37.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.344+0000 7f36f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.346 INFO:teuthology.orchestra.run.smithi083.stdout:81604378631 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 10 librados: watch_flush enter 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 10 librados: watch_flush exit 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 10 client.4277.objecter successfully canceled tick 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f36e806da70 msgr2=0x7f36e806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f36e806da70 0x7f36e806ff30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f36f8036000 tx=0x7f36f8030000 comp rx=0 tx=0).stop 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 10 monclient: shutdown 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 msgr2=0x7f3708135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708135c70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00b4a0 tx=0x7f36fc00b960 comp rx=0 tx=0).stop 2024-03-25T02:00:37.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 shutdown_connections 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f36e806da70 0x7f36e806ff30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3708140400 0x7f3708137300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f37080b2df0 0x7f3708135c70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 --2- 172.21.15.83:0/3652778816 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f37080b04a0 0x7f3708143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 >> 172.21.15.83:0/3652778816 conn(0x7f37080b8a00 msgr2=0x7f3708146b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 shutdown_connections 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 -- 172.21.15.83:0/3652778816 wait complete. 2024-03-25T02:00:37.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.346+0000 7f370c960640 1 librados: shutdown 2024-03-25T02:00:37.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.351+0000 7f6a20ff9640 1 -- 172.21.15.83:0/3506585645 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6a14021020 con 0x7f6a240b04a0 2024-03-25T02:00:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.351+0000 7f6a20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.351+0000 7f6a20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:37.359 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378632 got 81604378631 for osd.7 2024-03-25T02:00:37.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.394+0000 7f70a7a45640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:37.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.394+0000 7f70a7a45640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:37.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.394+0000 7f70a7a45640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:37.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.394+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f70a00af6f0 con 0x7f70a00b04a0 2024-03-25T02:00:37.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.397+0000 7f7095ffb640 1 -- 172.21.15.83:0/3320647035 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f70a00af6f0 con 0x7f70a00b04a0 2024-03-25T02:00:37.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.397+0000 7f7095ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:37.399 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.397+0000 7f7095ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.399 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-03-25T02:00:37.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 10 librados: watch_flush enter 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 10 librados: watch_flush exit 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 10 client.4347.objecter successfully canceled tick 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f708406da20 msgr2=0x7f708406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f708406da20 0x7f708406fee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7090036000 tx=0x7f709003b040 comp rx=0 tx=0).stop 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 10 monclient: shutdown 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70a00b04a0 msgr2=0x7f70a01437e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70a00b04a0 0x7f70a01437e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f709c00b420 tx=0x7f709c00b8e0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 shutdown_connections 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f708406da20 0x7f708406fee0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f70a0140400 0x7f70a01e58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f70a00b2df0 0x7f70a01318c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 --2- 172.21.15.83:0/3320647035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70a00b04a0 0x7f70a01437e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 >> 172.21.15.83:0/3320647035 conn(0x7f70a00b8a00 msgr2=0x7f70a0146b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 shutdown_connections 2024-03-25T02:00:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 -- 172.21.15.83:0/3320647035 wait complete. 2024-03-25T02:00:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.400+0000 7f70a7a45640 1 librados: shutdown 2024-03-25T02:00:37.412 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.4 2024-03-25T02:00:37.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.531+0000 7f6a2a96b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-25T02:00:37.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.532+0000 7f6a2a96b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-25T02:00:37.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.532+0000 7f6a2a96b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.532+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6a2414d8d0 con 0x7f6a240b04a0 2024-03-25T02:00:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.605+0000 7f74227fc640 1 -- 172.21.15.83:0/2587036759 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7428058a80 con 0x7f7424032150 2024-03-25T02:00:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.605+0000 7f74227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-25T02:00:37.606 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.605+0000 7f74227fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.607 INFO:teuthology.orchestra.run.smithi083.stdout:81604378633 2024-03-25T02:00:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.606+0000 7f7432dcf640 10 librados: watch_flush enter 2024-03-25T02:00:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 10 librados: watch_flush exit 2024-03-25T02:00:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 10 client.4363.objecter successfully canceled tick 2024-03-25T02:00:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f741006da50 msgr2=0x7f741006ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f741006da50 0x7f741006ff10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f742c0506f0 tx=0x7f742c06f040 comp rx=0 tx=0).stop 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 10 monclient: shutdown 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 msgr2=0x7f74240b0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240b0220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f742800c430 tx=0x7f742800c8f0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 shutdown_connections 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f741006da50 0x7f741006ff10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f7424034c10 0x7f74240b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7424032150 0x7f74240b0220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 --2- 172.21.15.83:0/2587036759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f742402fc90 0x7f74240c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 >> 172.21.15.83:0/2587036759 conn(0x7f7424005ed0 msgr2=0x7f74240373d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 shutdown_connections 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 -- 172.21.15.83:0/2587036759 wait complete. 2024-03-25T02:00:37.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.607+0000 7f7432dcf640 1 librados: shutdown 2024-03-25T02:00:37.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378633 for osd.6 2024-03-25T02:00:37.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.618+0000 7fd8fffff640 1 -- 172.21.15.83:0/833486044 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd91002ce20 con 0x7fd918147350 2024-03-25T02:00:37.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.618+0000 7fd8fffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-25T02:00:37.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.618+0000 7fd8fffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.620 INFO:teuthology.orchestra.run.smithi083.stdout:64424509445 2024-03-25T02:00:37.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.618+0000 7f9262ffd640 1 -- 172.21.15.83:0/2334861151 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f926404c060 con 0x7f92740aacf0 2024-03-25T02:00:37.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.619+0000 7f9262ffd640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f925406da50 0x7f925406ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:37.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.619+0000 7f927259c640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f925406da50 0x7f925406ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:37.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.619+0000 7f927259c640 10 monclient: get_auth_request con 0x7f925406da50 auth_method 0 2024-03-25T02:00:37.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.620+0000 7f9262ffd640 10 client.4384.objecter ms_handle_connect 0x7f925406da50 2024-03-25T02:00:37.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.620+0000 7f927259c640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f925406da50 0x7f925406ff10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f92680017f0 tx=0x7f9268009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:37.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 10 librados: watch_flush enter 2024-03-25T02:00:37.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 10 librados: watch_flush exit 2024-03-25T02:00:37.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 10 client.4369.objecter successfully canceled tick 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd90406da20 msgr2=0x7fd90406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd90406da20 0x7fd90406fee0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd9080011e0 tx=0x7fd908009040 comp rx=0 tx=0).stop 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 10 monclient: shutdown 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd918147350 msgr2=0x7fd91806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd918147350 0x7fd91806eb90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd91002c4d0 tx=0x7fd910004a40 comp rx=0 tx=0).stop 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 shutdown_connections 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd90406da20 0x7fd90406fee0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd918147350 0x7fd91806eb90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd9180b2df0 0x7fd918071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 --2- 172.21.15.83:0/833486044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd9180b04a0 0x7fd9181427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 >> 172.21.15.83:0/833486044 conn(0x7fd9180b8a00 msgr2=0x7fd9180b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 shutdown_connections 2024-03-25T02:00:37.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 -- 172.21.15.83:0/833486044 wait complete. 2024-03-25T02:00:37.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.621+0000 7fd91dd6b640 1 librados: shutdown 2024-03-25T02:00:37.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.622+0000 7f9262ffd640 1 -- 172.21.15.83:0/2334861151 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f9264078f10 con 0x7f92740aacf0 2024-03-25T02:00:37.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.622+0000 7f9262ffd640 10 client.4384.objecter ms_dispatch 0x7f9274000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:37.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.622+0000 7f9262ffd640 3 client.4384.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:37.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.622+0000 7f9262ffd640 3 client.4384.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:37.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.622+0000 7f9262ffd640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T02:00:37.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.3 2024-03-25T02:00:37.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.637+0000 7f9262ffd640 1 -- 172.21.15.83:0/2334861151 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9264040070 con 0x7f92740aacf0 2024-03-25T02:00:37.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.637+0000 7f9262ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:37.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.637+0000 7f9262ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.645+0000 7f6a20ff9640 1 -- 172.21.15.83:0/3506585645 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6a14059200 con 0x7f6a240b04a0 2024-03-25T02:00:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.645+0000 7f6a20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-25T02:00:37.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.645+0000 7f6a20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.647 INFO:teuthology.orchestra.run.smithi083.stdout:77309411333 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 10 librados: watch_flush enter 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 10 librados: watch_flush exit 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 10 client.4387.objecter successfully canceled tick 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6a1006da70 msgr2=0x7f6a1006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6a1006da70 0x7f6a1006ff30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6a18006fd0 tx=0x7f6a18008040 comp rx=0 tx=0).stop 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 10 monclient: shutdown 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a240b04a0 msgr2=0x7f6a24141d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a240b04a0 0x7f6a24141d20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400c400 tx=0x7f6a1400c8c0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 shutdown_connections 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6a1006da70 0x7f6a1006ff30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6a24147350 0x7f6a2413b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6a240b2df0 0x7f6a2413aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 --2- 172.21.15.83:0/3506585645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a240b04a0 0x7f6a24141d20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.647+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 >> 172.21.15.83:0/3506585645 conn(0x7f6a240b8a00 msgr2=0x7f6a24074f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.648+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 shutdown_connections 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.648+0000 7f6a2a96b640 1 -- 172.21.15.83:0/3506585645 wait complete. 2024-03-25T02:00:37.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.648+0000 7f6a2a96b640 1 librados: shutdown 2024-03-25T02:00:37.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411334 got 77309411333 for osd.5 2024-03-25T02:00:37.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.832+0000 7f927359e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-25T02:00:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.832+0000 7f927359e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-25T02:00:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.832+0000 7f927359e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:37.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.832+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f92740aa690 con 0x7f92740aacf0 2024-03-25T02:00:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.888+0000 7f9262ffd640 1 -- 172.21.15.83:0/2334861151 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9264031e90 con 0x7f92740aacf0 2024-03-25T02:00:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.888+0000 7f9262ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-03-25T02:00:37.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.888+0000 7f9262ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:85899345931 2024-03-25T02:00:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 10 librados: watch_flush enter 2024-03-25T02:00:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 10 librados: watch_flush exit 2024-03-25T02:00:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T02:00:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 10 client.4384.objecter successfully canceled tick 2024-03-25T02:00:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f925406da50 msgr2=0x7f925406ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f925406da50 0x7f925406ff10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f92680017f0 tx=0x7f9268009040 comp rx=0 tx=0).stop 2024-03-25T02:00:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 10 monclient: shutdown 2024-03-25T02:00:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92740aacf0 msgr2=0x7f92741d8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92740aacf0 0x7f92741d8910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f926402b8a0 tx=0x7f92640046c0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 shutdown_connections 2024-03-25T02:00:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f925406da50 0x7f925406ff10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.894 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9274140650 0x7f92741d8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92740aacf0 0x7f92741d8910 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 --2- 172.21.15.83:0/2334861151 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f92740a9ab0 0x7f9274146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 >> 172.21.15.83:0/2334861151 conn(0x7f92740b89f0 msgr2=0x7f92740b65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 shutdown_connections 2024-03-25T02:00:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 -- 172.21.15.83:0/2334861151 wait complete. 2024-03-25T02:00:37.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:37.890+0000 7f927359e640 1 librados: shutdown 2024-03-25T02:00:37.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345931 got 85899345931 for osd.1 2024-03-25T02:00:37.904 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:38.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-25T02:00:38.240 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-25T02:00:38.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.273+0000 7fa434414640 10 monclient: get_monmap_and_config 2024-03-25T02:00:38.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.273+0000 7fa434414640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.273+0000 7fa434414640 10 monclient: monmap: 2024-03-25T02:00:38.275 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.275 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.275 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:38.275079+0000 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:38.275079+0000 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:38.276 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 1 Processor -- start 2024-03-25T02:00:38.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 1 -- start start 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient: init 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa42c072660 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa42c1406d0 0x7fa42c06d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient(hunting): picked mon.noname-c con 0x7fa42c1406d0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c06daf0 0x7fa42c14da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient(hunting): picked mon.noname-b con 0x7fa42c06daf0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa42c072210 con 0x7fa42c072660 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fa42c072a40 con 0x7fa42c1406d0 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c06d190 con 0x7fa42c06daf0 2024-03-25T02:00:38.279 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.274+0000 7fa434414640 10 monclient(hunting): authenticate will time out at 1357.464937s 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4329b1640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c06daf0 0x7fa42c14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4319af640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa42c1406d0 0x7fa42c06d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4321b0640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fa42c072660 0x7fa42c13ffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45698/0 (socket says 172.21.15.83:45698) 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4321b0640 1 -- 172.21.15.83:0/3176173322 learned_addr learned my addr 172.21.15.83:0/3176173322 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4319af640 10 monclient(hunting): get_auth_request con 0x7fa42c1406d0 auth_method 0 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4319af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4319af640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4319af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.275+0000 7fa4329b1640 10 monclient(hunting): get_auth_request con 0x7fa42c06daf0 auth_method 0 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4329b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.280 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4329b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4329b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4319af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4319af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4319af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4329b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4329b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.281 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.276+0000 7fa4329b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.277+0000 7fa4319af640 10 monclient(hunting): handle_auth_done global_id 4393 payload 274 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.277+0000 7fa4319af640 1 -- 172.21.15.83:0/3176173322 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c06daf0 msgr2=0x7fa42c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.278+0000 7fa4319af640 1 --2- 172.21.15.83:0/3176173322 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c06daf0 0x7fa42c14da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.278+0000 7fa4319af640 1 -- 172.21.15.83:0/3176173322 >> v1:172.21.15.83:6789/0 conn(0x7fa42c072660 legacy=0x7fa42c13ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.278+0000 7fa4319af640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.278+0000 7fa4319af640 1 monclient: found mon.noname-c 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.278+0000 7fa4319af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.278+0000 7fa4319af640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.282 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.278+0000 7fa4319af640 1 -- 172.21.15.83:0/3176173322 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c140c40 con 0x7fa42c1406d0 2024-03-25T02:00:38.283 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.280+0000 7fa4319af640 1 --2- 172.21.15.83:0/3176173322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c1406d0 0x7fa42c06d5b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0097a0 tx=0x7fa41c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c05439beb1f0f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 1 -- 172.21.15.83:0/3176173322 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c0315a0 con 0x7fa42c1406d0 2024-03-25T02:00:38.306 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:38.307 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: dump: 2024-03-25T02:00:38.307 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.307 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.307 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 1 -- 172.21.15.83:0/3176173322 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41c037070 con 0x7fa42c1406d0 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 1 -- 172.21.15.83:0/3176173322 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c031960 con 0x7fa42c1406d0 2024-03-25T02:00:38.308 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa434414640 5 monclient: authenticate success, global_id 4393 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa434414640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa434414640 4 set_mon_vals no callback set 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.304+0000 7fa4311ae640 10 monclient: dump: 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.309 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 10 monclient: shutdown 2024-03-25T02:00:38.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.312 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 -- 172.21.15.83:0/3176173322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c1406d0 msgr2=0x7fa42c06d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 --2- 172.21.15.83:0/3176173322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c1406d0 0x7fa42c06d5b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0097a0 tx=0x7fa41c02b840 comp rx=0 tx=0).stop 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 -- 172.21.15.83:0/3176173322 shutdown_connections 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 --2- 172.21.15.83:0/3176173322 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c06daf0 0x7fa42c14da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 --2- 172.21.15.83:0/3176173322 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c1406d0 0x7fa42c06d5b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 -- 172.21.15.83:0/3176173322 >> 172.21.15.83:0/3176173322 conn(0x7fa42c067e90 msgr2=0x7fa42c0b69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 -- 172.21.15.83:0/3176173322 shutdown_connections 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.305+0000 7fa434414640 1 -- 172.21.15.83:0/3176173322 wait complete. 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.306+0000 7fa434414640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.306+0000 7fa434414640 10 monclient: monmap: 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:38.313 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.307+0000 7fa434414640 1 librados: starting msgr at 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.307+0000 7fa434414640 1 librados: starting objecter 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.307+0000 7fa434414640 1 Processor -- start 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.307+0000 7fa434414640 1 -- start start 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.307+0000 7fa434414640 1 librados: setting wanted keys 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.307+0000 7fa434414640 1 librados: calling monclient init 2024-03-25T02:00:38.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.307+0000 7fa434414640 10 monclient: init 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c06daf0 0x7fa42c0adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): picked mon.noname-b con 0x7fa42c06daf0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c072660 0x7fa42c1d87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): picked mon.noname-a con 0x7fa42c072660 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c1406d0 0x7fa42c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): picked mon.noname-c con 0x7fa42c1406d0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c06e090 con 0x7fa42c072660 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c06df10 con 0x7fa42c06daf0 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c0b0e80 con 0x7fa42c1406d0 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa434414640 10 monclient(hunting): authenticate will time out at 1357.498182s 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa4329b1640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c1406d0 0x7fa42c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.308+0000 7fa4321b0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c06daf0 0x7fa42c0adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c1406d0 0x7fa42c0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54618/0 (socket says 172.21.15.83:54618) 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 1 -- 172.21.15.83:0/59789742 learned_addr learned my addr 172.21.15.83:0/59789742 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 10 monclient(hunting): get_auth_request con 0x7fa42c1406d0 auth_method 0 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4321b0640 10 monclient(hunting): get_auth_request con 0x7fa42c06daf0 auth_method 0 2024-03-25T02:00:38.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4321b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4321b0640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4321b0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4321b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4321b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4321b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.309+0000 7fa4329b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-03-25T02:00:38.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 1 -- 172.21.15.83:0/59789742 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c1406d0 msgr2=0x7fa42c0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c1406d0 0x7fa42c0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 1 -- 172.21.15.83:0/59789742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c072660 msgr2=0x7fa42c1d87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c072660 0x7fa42c1d87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 1 monclient: found mon.noname-b 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.312+0000 7fa4321b0640 1 -- 172.21.15.83:0/59789742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c0ae120 con 0x7fa42c06daf0 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.314+0000 7fa4321b0640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c06daf0 0x7fa42c0adbe0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c370 tx=0x7fa42800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.314+0000 7fa422ffd640 10 client.?.objecter ms_handle_connect 0x7fa42c06daf0 2024-03-25T02:00:38.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.314+0000 7fa422ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:38.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 1 -- 172.21.15.83:0/59789742 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428017070 con 0x7fa42c06daf0 2024-03-25T02:00:38.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:38.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: dump: 2024-03-25T02:00:38.320 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.320 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.320 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 1 -- 172.21.15.83:0/59789742 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa428005ce0 con 0x7fa42c06daf0 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 5 monclient: authenticate success, global_id 4396 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 10 monclient: _renew_subs 2024-03-25T02:00:38.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c0b0900 con 0x7fa42c06daf0 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 1 -- 172.21.15.83:0/59789742 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa428007310 con 0x7fa42c06daf0 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa422ffd640 10 monclient: dump: 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.323 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.324 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa423fff640 4 set_mon_vals no callback set 2024-03-25T02:00:38.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:38.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 10 monclient: _renew_subs 2024-03-25T02:00:38.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c147450 con 0x7fa42c06daf0 2024-03-25T02:00:38.324 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.318+0000 7fa434414640 1 librados: init done 2024-03-25T02:00:38.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.319+0000 7fa434414640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.319+0000 7fa434414640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.319+0000 7fa434414640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.319+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c075810 con 0x7fa42c06daf0 2024-03-25T02:00:38.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.338+0000 7fa422ffd640 1 -- 172.21.15.83:0/59789742 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fa42801a070 con 0x7fa42c06daf0 2024-03-25T02:00:38.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.338+0000 7fa422ffd640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa41006da70 0x7fa41006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.340 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.338+0000 7fa4319af640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa41006da70 0x7fa41006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.338+0000 7fa4319af640 10 monclient: get_auth_request con 0x7fa41006da70 auth_method 0 2024-03-25T02:00:38.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.339+0000 7fa4319af640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa41006da70 0x7fa41006ff30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa41c036000 tx=0x7fa41c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.341 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.339+0000 7fa422ffd640 10 client.4396.objecter ms_handle_connect 0x7fa41006da70 2024-03-25T02:00:38.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.344+0000 7f6b948c5640 10 monclient: get_monmap_and_config 2024-03-25T02:00:38.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.344+0000 7f6b948c5640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.344+0000 7f6b948c5640 10 monclient: monmap: 2024-03-25T02:00:38.346 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.346 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:38.345504+0000 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:38.345504+0000 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:38.347 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:38.348 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.344+0000 7f6b948c5640 1 Processor -- start 2024-03-25T02:00:38.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.344+0000 7f6b948c5640 1 -- start start 2024-03-25T02:00:38.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.344+0000 7f6b948c5640 10 monclient: init 2024-03-25T02:00:38.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b900b04a0 0x7f6b90146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient(hunting): picked mon.noname-b con 0x7f6b900b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6b900b2df0 0x7f6b90147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient(hunting): picked mon.noname-a con 0x7f6b900b2df0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b90140400 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b900b2410 con 0x7f6b90140400 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6b900b26f0 con 0x7f6b900b2df0 2024-03-25T02:00:38.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f6b900b29d0 con 0x7f6b900b04a0 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b948c5640 10 monclient(hunting): authenticate will time out at 1357.535401s 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.345+0000 7f6b8dd9b640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6b900b2df0 0x7f6b90147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b900b04a0 0x7f6b90146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b900b04a0 0x7f6b90146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54632/0 (socket says 172.21.15.83:54632) 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 1 -- 172.21.15.83:0/3920414052 learned_addr learned my addr 172.21.15.83:0/3920414052 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 10 monclient(hunting): get_auth_request con 0x7f6b900b04a0 auth_method 0 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8dd9b640 10 monclient(hunting): get_auth_request con 0x7f6b900b2df0 auth_method 0 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.346+0000 7f6b8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.347+0000 7f6b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.347+0000 7f6b8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.347+0000 7f6b8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 1 -- 172.21.15.83:0/3920414052 >> v2:172.21.15.83:3300/0 conn(0x7f6b900b2df0 msgr2=0x7f6b90147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 1 --2- 172.21.15.83:0/3920414052 >> v2:172.21.15.83:3300/0 conn(0x7f6b900b2df0 0x7f6b90147180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 1 -- 172.21.15.83:0/3920414052 >> v1:172.21.15.83:6789/0 conn(0x7f6b90140400 legacy=0x7f6b90140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 1 monclient: found mon.noname-b 2024-03-25T02:00:38.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.349+0000 7f6b8e59c640 1 -- 172.21.15.83:0/3920414052 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b900b3380 con 0x7f6b900b04a0 2024-03-25T02:00:38.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.351+0000 7f6b8e59c640 1 --2- 172.21.15.83:0/3920414052 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b900b04a0 0x7f6b90146c40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009e50 tx=0x7f6b8802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8de330404598c28 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b8d59a640 1 -- 172.21.15.83:0/3920414052 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8802cb90 con 0x7f6b900b04a0 2024-03-25T02:00:38.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.355 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b8d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b8d59a640 10 monclient: dump: 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b8d59a640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.356 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b8d59a640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b948c5640 5 monclient: authenticate success, global_id 4289 2024-03-25T02:00:38.357 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.353+0000 7f6b948c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b8d59a640 1 -- 172.21.15.83:0/3920414052 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8802cd30 con 0x7f6b900b04a0 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b8d59a640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b8d59a640 1 -- 172.21.15.83:0/3920414052 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88034610 con 0x7f6b900b04a0 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b8d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b8d59a640 10 monclient: dump: 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.358 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b948c5640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.355+0000 7f6b948c5640 4 set_mon_vals no callback set 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 10 monclient: shutdown 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 -- 172.21.15.83:0/3920414052 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b900b04a0 msgr2=0x7f6b90146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 --2- 172.21.15.83:0/3920414052 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b900b04a0 0x7f6b90146c40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009e50 tx=0x7f6b8802bff0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 -- 172.21.15.83:0/3920414052 shutdown_connections 2024-03-25T02:00:38.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 --2- 172.21.15.83:0/3920414052 >> v2:172.21.15.83:3300/0 conn(0x7f6b900b2df0 0x7f6b90147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 --2- 172.21.15.83:0/3920414052 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b900b04a0 0x7f6b90146c40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 -- 172.21.15.83:0/3920414052 >> 172.21.15.83:0/3920414052 conn(0x7f6b900b8a00 msgr2=0x7f6b900b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 -- 172.21.15.83:0/3920414052 shutdown_connections 2024-03-25T02:00:38.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.356+0000 7f6b948c5640 1 -- 172.21.15.83:0/3920414052 wait complete. 2024-03-25T02:00:38.360 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 10 monclient: monmap: 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.363 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 1 librados: starting msgr at 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 1 librados: starting objecter 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 1 Processor -- start 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 1 -- start start 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 1 librados: setting wanted keys 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 1 librados: calling monclient init 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.357+0000 7f6b948c5640 10 monclient: init 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b900b04a0 0x7f6b90143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.364 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): picked mon.noname-a con 0x7f6b900b04a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b900b2df0 0x7f6b901dbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): picked mon.noname-b con 0x7f6b900b2df0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b90140400 0x7f6b901dc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): picked mon.noname-c con 0x7f6b90140400 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9014dc70 con 0x7f6b900b04a0 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9014daf0 con 0x7f6b900b2df0 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9014ddf0 con 0x7f6b90140400 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b948c5640 10 monclient(hunting): authenticate will time out at 1357.548304s 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b90140400 0x7f6b901dc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.358+0000 7f6b8e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b900b04a0 0x7f6b90143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b90140400 0x7f6b901dc240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54638/0 (socket says 172.21.15.83:54638) 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 1 -- 172.21.15.83:0/1738840272 learned_addr learned my addr 172.21.15.83:0/1738840272 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 10 monclient(hunting): get_auth_request con 0x7f6b90140400 auth_method 0 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8e59c640 10 monclient(hunting): get_auth_request con 0x7f6b900b04a0 auth_method 0 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8e59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.359+0000 7f6b8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.360+0000 7f6b8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.360+0000 7f6b8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.360+0000 7f6b8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 1 -- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b900b2df0 msgr2=0x7f6b901dbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b900b2df0 0x7f6b901dbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 1 -- 172.21.15.83:0/1738840272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b900b04a0 msgr2=0x7f6b90143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b900b04a0 0x7f6b90143ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 1 monclient: found mon.noname-c 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.364+0000 7f6b8ed9d640 1 -- 172.21.15.83:0/1738840272 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9013a560 con 0x7f6b90140400 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.366+0000 7f6b8ed9d640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b90140400 0x7f6b901dc240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000c3c0 tx=0x7f6b8000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.367+0000 7f6b7effd640 10 client.?.objecter ms_handle_connect 0x7f6b90140400 2024-03-25T02:00:38.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.367+0000 7f6b7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 1 -- 172.21.15.83:0/1738840272 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80017070 con 0x7f6b90140400 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: dump: 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.373 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 1 -- 172.21.15.83:0/1738840272 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b800052d0 con 0x7f6b90140400 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 1 -- 172.21.15.83:0/1738840272 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80003830 con 0x7f6b90140400 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b7effd640 10 monclient: dump: 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.374 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 5 monclient: authenticate success, global_id 4292 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b8cd99640 4 set_mon_vals no callback set 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 10 monclient: _renew_subs 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b90141000 con 0x7f6b90140400 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 10 monclient: _renew_subs 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b901dc960 con 0x7f6b90140400 2024-03-25T02:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.370+0000 7f6b948c5640 1 librados: init done 2024-03-25T02:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.371+0000 7f6b948c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.372+0000 7f6b948c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.372+0000 7f6b948c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.372+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b90139370 con 0x7f6b90140400 2024-03-25T02:00:38.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.393+0000 7f6b7effd640 1 -- 172.21.15.83:0/1738840272 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f6b8001a070 con 0x7f6b90140400 2024-03-25T02:00:38.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.393+0000 7f6b7effd640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6b7006da20 0x7f6b7006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.393+0000 7f6b8e59c640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6b7006da20 0x7f6b7006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.393+0000 7f6b8e59c640 10 monclient: get_auth_request con 0x7f6b7006da20 auth_method 0 2024-03-25T02:00:38.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.394+0000 7f6b7effd640 10 client.4292.objecter ms_handle_connect 0x7f6b7006da20 2024-03-25T02:00:38.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.394+0000 7f6b8e59c640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6b7006da20 0x7f6b7006fee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6b88035000 tx=0x7f6b88030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.396+0000 7f6b7effd640 1 -- 172.21.15.83:0/1738840272 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f6b80056ad0 con 0x7f6b90140400 2024-03-25T02:00:38.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.396+0000 7f6b7effd640 10 client.4292.objecter ms_dispatch 0x7f6b90000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:38.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.396+0000 7f6b7effd640 3 client.4292.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:38.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.396+0000 7f6b7effd640 3 client.4292.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:38.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.396+0000 7f6b7effd640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T02:00:38.413 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.464+0000 7f210fd79640 10 monclient: get_monmap_and_config 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.464+0000 7f210fd79640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.464+0000 7f210fd79640 10 monclient: monmap: 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:38.465990+0000 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:38.465990+0000 2024-03-25T02:00:38.466 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.467 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.467 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:38.467 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:38.467 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:38.467 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:38.467 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:38.467 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 Processor -- start 2024-03-25T02:00:38.468 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 -- start start 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient: init 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21080a9b70 0x7f21081437c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): picked mon.noname-a con 0x7f21080a9b70 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080b33f0 0x7f2108140320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): picked mon.noname-b con 0x7f21080b33f0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f21080a9f50 0x7f2108140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): picked mon.noname-c con 0x7f21080a9f50 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f21080abae0 con 0x7f21080a9b70 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f21080abdc0 con 0x7f21080a9f50 2024-03-25T02:00:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f21080ac0a0 con 0x7f21080b33f0 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.465+0000 7f210fd79640 10 monclient(hunting): authenticate will time out at 1357.655934s 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.466+0000 7f210db15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21080a9b70 0x7f21081437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f21080a9b70 0x7f21081437c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56140/0 (socket says 172.21.15.83:56140) 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 1 -- 172.21.15.83:0/4150892899 learned_addr learned my addr 172.21.15.83:0/4150892899 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 10 monclient(hunting): get_auth_request con 0x7f21080a9b70 auth_method 0 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.467+0000 7f210db15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 1 -- 172.21.15.83:0/4150892899 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080b33f0 msgr2=0x7f2108140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 1 --2- 172.21.15.83:0/4150892899 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080b33f0 0x7f2108140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 1 -- 172.21.15.83:0/4150892899 >> v2:172.21.15.153:3300/0 conn(0x7f21080a9f50 msgr2=0x7f2108140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 1 --2- 172.21.15.83:0/4150892899 >> v2:172.21.15.153:3300/0 conn(0x7f21080a9f50 0x7f2108140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 1 monclient: found mon.noname-a 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.472 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.470+0000 7f210db15640 1 -- 172.21.15.83:0/4150892899 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21081432a0 con 0x7f21080a9b70 2024-03-25T02:00:38.474 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.473+0000 7f210db15640 1 --2- 172.21.15.83:0/4150892899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080a9b70 0x7f21081437c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f21080b4b70 tx=0x7f210402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82eb33c5337d17c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210cb13640 1 -- 172.21.15.83:0/4150892899 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104035020 con 0x7f21080a9b70 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210cb13640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210cb13640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210cb13640 10 monclient: dump: 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.477 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.478 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210cb13640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210cb13640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210cb13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210fd79640 5 monclient: authenticate success, global_id 4362 2024-03-25T02:00:38.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.475+0000 7f210fd79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:38.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210cb13640 1 -- 172.21.15.83:0/4150892899 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2104031870 con 0x7f21080a9b70 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210cb13640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210cb13640 1 -- 172.21.15.83:0/4150892899 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104031c40 con 0x7f21080a9b70 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210cb13640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210cb13640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210cb13640 10 monclient: dump: 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.479 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.480 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.480 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.480 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210fd79640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:38.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.476+0000 7f210fd79640 4 set_mon_vals no callback set 2024-03-25T02:00:38.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 10 monclient: shutdown 2024-03-25T02:00:38.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 1 -- 172.21.15.83:0/4150892899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080a9b70 msgr2=0x7f21081437c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 1 --2- 172.21.15.83:0/4150892899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080a9b70 0x7f21081437c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f21080b4b70 tx=0x7f210402b910 comp rx=0 tx=0).stop 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 1 -- 172.21.15.83:0/4150892899 shutdown_connections 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 1 --2- 172.21.15.83:0/4150892899 >> v2:172.21.15.153:3300/0 conn(0x7f21080a9f50 0x7f2108140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 1 --2- 172.21.15.83:0/4150892899 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080b33f0 0x7f2108140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 1 --2- 172.21.15.83:0/4150892899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080a9b70 0x7f21081437c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.477+0000 7f210fd79640 1 -- 172.21.15.83:0/4150892899 >> 172.21.15.83:0/4150892899 conn(0x7f2108067e30 msgr2=0x7f21080b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.478+0000 7f210fd79640 1 -- 172.21.15.83:0/4150892899 shutdown_connections 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.478+0000 7f210fd79640 1 -- 172.21.15.83:0/4150892899 wait complete. 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.478+0000 7f210fd79640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.482 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 10 monclient: monmap: 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 1 librados: starting msgr at 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 1 librados: starting objecter 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 1 Processor -- start 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 1 -- start start 2024-03-25T02:00:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 1 librados: setting wanted keys 2024-03-25T02:00:38.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 1 librados: calling monclient init 2024-03-25T02:00:38.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.479+0000 7f210fd79640 10 monclient: init 2024-03-25T02:00:38.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:38.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080a9b70 0x7f21081cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): picked mon.noname-c con 0x7f21080a9b70 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21080a9f50 0x7f21081ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): picked mon.noname-b con 0x7f21080a9f50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080b33f0 0x7f21081cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): picked mon.noname-a con 0x7f21080b33f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2108146680 con 0x7f21080b33f0 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2108143f10 con 0x7f21080a9f50 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2108146800 con 0x7f21080a9b70 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.480+0000 7f210fd79640 10 monclient(hunting): authenticate will time out at 1357.670543s 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.481+0000 7f210e316640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080b33f0 0x7f21081cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.481+0000 7f210e316640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080b33f0 0x7f21081cd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56156/0 (socket says 172.21.15.83:56156) 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.481+0000 7f210e316640 1 -- 172.21.15.83:0/121833758 learned_addr learned my addr 172.21.15.83:0/121833758 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.481+0000 7f210e316640 10 monclient(hunting): get_auth_request con 0x7f21080b33f0 auth_method 0 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.481+0000 7f210e316640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.481+0000 7f210e316640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.481+0000 7f210e316640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.482+0000 7f210e316640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.482+0000 7f210e316640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.482+0000 7f210e316640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.484+0000 7f210e316640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 1 -- 172.21.15.83:0/121833758 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080a9b70 msgr2=0x7f21081cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080a9b70 0x7f21081cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 1 -- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21080a9f50 msgr2=0x7f21081ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21080a9f50 0x7f21081ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 1 monclient: found mon.noname-a 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.485+0000 7f210e316640 1 -- 172.21.15.83:0/121833758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21081cef20 con 0x7f21080b33f0 2024-03-25T02:00:38.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.487+0000 7f210e316640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080b33f0 0x7f21081cd320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f210000b9e0 tx=0x7f210000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.487+0000 7f20fe7fc640 10 client.?.objecter ms_handle_connect 0x7f21080b33f0 2024-03-25T02:00:38.489 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.487+0000 7f20fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:38.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 1 -- 172.21.15.83:0/121833758 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100019070 con 0x7f21080b33f0 2024-03-25T02:00:38.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:38.494 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 10 monclient: dump: 2024-03-25T02:00:38.495 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.495 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.495 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.495 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.495 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.495 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f20fe7fc640 1 -- 172.21.15.83:0/121833758 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210000cae0 con 0x7f21080b33f0 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.490+0000 7f210fd79640 5 monclient: authenticate success, global_id 4365 2024-03-25T02:00:38.496 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 monclient: _renew_subs 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21081cfed0 con 0x7f21080b33f0 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f20fe7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f20fe7fc640 1 -- 172.21.15.83:0/121833758 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100014440 con 0x7f21080b33f0 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f20fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f20fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f20fe7fc640 10 monclient: dump: 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.497 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f20fffff640 4 set_mon_vals no callback set 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 monclient: _renew_subs 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21081cda40 con 0x7f21080b33f0 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 1 librados: init done 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.498 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.491+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2108075810 con 0x7f21080b33f0 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.515+0000 7f679ec08640 10 monclient: get_monmap_and_config 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.515+0000 7f679ec08640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.515+0000 7f679ec08640 10 monclient: monmap: 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:38.516948+0000 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:38.516948+0000 2024-03-25T02:00:38.517 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.518 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.518 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:38.518 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:38.518 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:38.518 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:38.518 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:38.518 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 1 Processor -- start 2024-03-25T02:00:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 1 -- start start 2024-03-25T02:00:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient: init 2024-03-25T02:00:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-03-25T02:00:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6798070e80 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:38.519 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6798072e20 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6798148e50 0x7f679814b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient(hunting): picked mon.noname-c con 0x7f6798148e50 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6798070810 con 0x7f6798072e20 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f679813fc50 con 0x7f6798070e80 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f6798071260 con 0x7f6798148e50 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.520 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.516+0000 7f679ec08640 10 monclient(hunting): authenticate will time out at 1357.706862s 2024-03-25T02:00:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.568+0000 7fa422ffd640 1 -- 172.21.15.83:0/59789742 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fa428056a90 con 0x7fa42c06daf0 2024-03-25T02:00:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.568+0000 7fa422ffd640 10 client.4396.objecter ms_dispatch 0x7fa42c000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.568+0000 7fa422ffd640 3 client.4396.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.568+0000 7fa422ffd640 3 client.4396.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:38.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.568+0000 7fa422ffd640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T02:00:38.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.578+0000 7fa422ffd640 1 -- 172.21.15.83:0/59789742 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa428021020 con 0x7fa42c06daf0 2024-03-25T02:00:38.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.578+0000 7fa422ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.578+0000 7fa422ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:38.589 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.588+0000 7f679c9a4640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6798070e80 0x7f679813f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33366/0 (socket says 172.21.15.83:33366) 2024-03-25T02:00:38.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.588+0000 7f679c9a4640 1 -- 172.21.15.83:0/1745807282 learned_addr learned my addr 172.21.15.83:0/1745807282 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:38.590 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.589+0000 7f679d1a5640 1 --2- 172.21.15.83:0/1745807282 >> v2:172.21.15.153:3300/0 conn(0x7f6798148e50 0x7f679814b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.589+0000 7f679d1a5640 10 monclient(hunting): get_auth_request con 0x7f6798148e50 auth_method 0 2024-03-25T02:00:38.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.589+0000 7f679d1a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.589+0000 7f679d1a5640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.591 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.589+0000 7f679d1a5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.592 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.591+0000 7f679d1a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.591+0000 7f679d1a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.593 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.591+0000 7f679d1a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.593+0000 7f67977fe640 1 -- 172.21.15.83:0/1745807282 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040535431 0 0) 0x7f679813fc50 con 0x7f6798070e80 2024-03-25T02:00:38.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.593+0000 7f67977fe640 10 monclient(hunting): _negotiate 2024-03-25T02:00:38.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.593+0000 7f67977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.593+0000 7f67977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.593+0000 7f67977fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T02:00:38.595 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.593+0000 7f67977fe640 1 -- 172.21.15.83:0/1745807282 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6788003520 con 0x7f6798070e80 2024-03-25T02:00:38.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2024-03-25T02:00:38.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 1 -- 172.21.15.83:0/1745807282 >> v1:172.21.15.153:6789/0 conn(0x7f6798070e80 legacy=0x7f679813f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 1 -- 172.21.15.83:0/1745807282 >> v1:172.21.15.83:6789/0 conn(0x7f6798072e20 legacy=0x7f679813c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 1 monclient: found mon.noname-c 2024-03-25T02:00:38.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.594+0000 7f679d1a5640 1 -- 172.21.15.83:0/1745807282 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67980aeee0 con 0x7f6798148e50 2024-03-25T02:00:38.600 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.598+0000 7f679d1a5640 1 --2- 172.21.15.83:0/1745807282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798148e50 0x7f679814b240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f679000aea0 tx=0x7f679002cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4a5517db07f7cda server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.603 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 1 -- 172.21.15.83:0/1745807282 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679002dac0 con 0x7f6798148e50 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: dump: 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 1 -- 172.21.15.83:0/1745807282 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f679002dc60 con 0x7f6798148e50 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 1 -- 172.21.15.83:0/1745807282 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6790008660 con 0x7f6798148e50 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 5 monclient: authenticate success, global_id 4399 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 4 set_mon_vals no callback set 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f67977fe640 10 monclient: dump: 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.605 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.606 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 10 monclient: shutdown 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 1 -- 172.21.15.83:0/1745807282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798148e50 msgr2=0x7f679814b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 1 --2- 172.21.15.83:0/1745807282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798148e50 0x7f679814b240 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f679000aea0 tx=0x7f679002cf20 comp rx=0 tx=0).stop 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 1 -- 172.21.15.83:0/1745807282 shutdown_connections 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 1 --2- 172.21.15.83:0/1745807282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798148e50 0x7f679814b240 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 1 -- 172.21.15.83:0/1745807282 >> 172.21.15.83:0/1745807282 conn(0x7f67980b4a10 msgr2=0x7f67980b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 1 -- 172.21.15.83:0/1745807282 shutdown_connections 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.602+0000 7f679ec08640 1 -- 172.21.15.83:0/1745807282 wait complete. 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.609 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 10 monclient: monmap: 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 1 librados: starting msgr at 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 1 librados: starting objecter 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 1 Processor -- start 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 1 -- start start 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 1 librados: setting wanted keys 2024-03-25T02:00:38.610 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 1 librados: calling monclient init 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.603+0000 7f679ec08640 10 monclient: init 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798070e80 0x7f67980adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): picked mon.noname-b con 0x7f6798070e80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6798072e20 0x7f67981d8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): picked mon.noname-c con 0x7f6798072e20 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6798148e50 0x7f67981408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): picked mon.noname-a con 0x7f6798148e50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67980725d0 con 0x7f6798148e50 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67980722f0 con 0x7f6798070e80 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.611 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f6798070510 con 0x7f6798072e20 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.604+0000 7f679ec08640 10 monclient(hunting): authenticate will time out at 1357.794641s 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.605+0000 7f679c9a4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798070e80 0x7f67980adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.605+0000 7f679c9a4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798070e80 0x7f67980adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40798/0 (socket says 172.21.15.83:40798) 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.605+0000 7f679c9a4640 1 -- 172.21.15.83:0/2636188064 learned_addr learned my addr 172.21.15.83:0/2636188064 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.605+0000 7f679c9a4640 10 monclient(hunting): get_auth_request con 0x7f6798070e80 auth_method 0 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.605+0000 7f679c9a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.605+0000 7f679c9a4640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.605+0000 7f679c9a4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.606+0000 7f679c9a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.606+0000 7f679c9a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.606+0000 7f679c9a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 1 -- 172.21.15.83:0/2636188064 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6798072e20 msgr2=0x7f67981d8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6798072e20 0x7f67981d8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 1 -- 172.21.15.83:0/2636188064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6798148e50 msgr2=0x7f67981408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6798148e50 0x7f67981408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 1 monclient: found mon.noname-b 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.613 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.609+0000 7f679c9a4640 1 -- 172.21.15.83:0/2636188064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67981dbd90 con 0x7f6798070e80 2024-03-25T02:00:38.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.611+0000 7f679c9a4640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798070e80 0x7f67980adb60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f678c00c530 tx=0x7f678c00c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.611+0000 7f67957fa640 10 client.?.objecter ms_handle_connect 0x7f6798070e80 2024-03-25T02:00:38.614 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.611+0000 7f67957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:38.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 1 -- 172.21.15.83:0/2636188064 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c017070 con 0x7f6798070e80 2024-03-25T02:00:38.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.617 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: dump: 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.618 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 5 monclient: authenticate success, global_id 4405 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 1 -- 172.21.15.83:0/2636188064 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678c005ce0 con 0x7f6798070e80 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 10 monclient: _renew_subs 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6798148bd0 con 0x7f6798070e80 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 1 -- 172.21.15.83:0/2636188064 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c0072b0 con 0x7f6798070e80 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f67957fa640 10 monclient: dump: 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.619 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 10 monclient: _renew_subs 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67981dc2b0 con 0x7f6798070e80 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f679ec08640 1 librados: init done 2024-03-25T02:00:38.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.615+0000 7f6796ffd640 4 set_mon_vals no callback set 2024-03-25T02:00:38.621 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-25T02:00:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.616+0000 7f679ec08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.616+0000 7f679ec08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.616+0000 7f679ec08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.616+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6798075810 con 0x7f6798070e80 2024-03-25T02:00:38.634 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-25T02:00:38.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.637+0000 7f67957fa640 1 -- 172.21.15.83:0/2636188064 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f678c01a070 con 0x7f6798070e80 2024-03-25T02:00:38.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.638+0000 7f67957fa640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677c06da70 0x7f677c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.638+0000 7f6797fff640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677c06da70 0x7f677c06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.638+0000 7f6797fff640 10 monclient: get_auth_request con 0x7f677c06da70 auth_method 0 2024-03-25T02:00:38.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.639+0000 7f67957fa640 10 client.4405.objecter ms_handle_connect 0x7f677c06da70 2024-03-25T02:00:38.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.639+0000 7f6797fff640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677c06da70 0x7f677c06ff30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f679813bf60 tx=0x7f6784007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.640+0000 7f67957fa640 1 -- 172.21.15.83:0/2636188064 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f678c056bc0 con 0x7f6798070e80 2024-03-25T02:00:38.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.640+0000 7f67957fa640 10 client.4405.objecter ms_dispatch 0x7f6798000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:38.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.640+0000 7f67957fa640 3 client.4405.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:38.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.640+0000 7f67957fa640 3 client.4405.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:38.643 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.640+0000 7f67957fa640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T02:00:38.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-25T02:00:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.677+0000 7f6b7effd640 1 -- 172.21.15.83:0/1738840272 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6b90139370 con 0x7f6b90140400 2024-03-25T02:00:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.677+0000 7f6b7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.677+0000 7f6b7effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:38.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.726+0000 7f20fe7fc640 1 -- 172.21.15.83:0/121833758 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f210001c070 con 0x7f21080b33f0 2024-03-25T02:00:38.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.726+0000 7f20fe7fc640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f20ec06da20 0x7f20ec06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.727+0000 7f210db15640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f20ec06da20 0x7f20ec06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.729 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.727+0000 7f210db15640 10 monclient: get_auth_request con 0x7f20ec06da20 auth_method 0 2024-03-25T02:00:38.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.727+0000 7f210db15640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f20ec06da20 0x7f20ec06fee0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2104036000 tx=0x7f2104030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.727+0000 7f20fe7fc640 10 client.4365.objecter ms_handle_connect 0x7f20ec06da20 2024-03-25T02:00:38.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.760+0000 7f272a124640 10 monclient: get_monmap_and_config 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.760+0000 7f272a124640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.760+0000 7f272a124640 10 monclient: monmap: 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:38.761483+0000 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:38.761483+0000 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:38.763 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:38.763 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.761+0000 7f272a124640 1 Processor -- start 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.761+0000 7f272a124640 1 -- start start 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.761+0000 7f272a124640 10 monclient: init 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27240b04a0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f27240b2df0 0x7f2724140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient(hunting): picked mon.noname-a con 0x7f27240b2df0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2724147350 0x7f2724140940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient(hunting): picked mon.noname-b con 0x7f2724147350 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27240b2410 con 0x7f27240b04a0 2024-03-25T02:00:38.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f2724140e80 con 0x7f27240b2df0 2024-03-25T02:00:38.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f27240b26f0 con 0x7f2724147350 2024-03-25T02:00:38.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.762+0000 7f272a124640 10 monclient(hunting): authenticate will time out at 1357.952310s 2024-03-25T02:00:38.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.763+0000 7f2723fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2724147350 0x7f2724140940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.763+0000 7f2723fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2724147350 0x7f2724140940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54664/0 (socket says 172.21.15.83:54664) 2024-03-25T02:00:38.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.763+0000 7f2723fff640 1 -- 172.21.15.83:0/1935714113 learned_addr learned my addr 172.21.15.83:0/1935714113 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.763+0000 7f2723fff640 10 monclient(hunting): get_auth_request con 0x7f2724147350 auth_method 0 2024-03-25T02:00:38.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.763+0000 7f2723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.763+0000 7f2723fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.763+0000 7f2723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.764+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.764+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.764+0000 7f2723fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 1 -- 172.21.15.83:0/1935714113 >> v2:172.21.15.83:3300/0 conn(0x7f27240b2df0 msgr2=0x7f2724140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 1 --2- 172.21.15.83:0/1935714113 >> v2:172.21.15.83:3300/0 conn(0x7f27240b2df0 0x7f2724140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 1 -- 172.21.15.83:0/1935714113 >> v1:172.21.15.153:6789/0 conn(0x7f27240b04a0 legacy=0x7f2724146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 1 monclient: found mon.noname-b 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.766+0000 7f2723fff640 1 -- 172.21.15.83:0/1935714113 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27240b29d0 con 0x7f2724147350 2024-03-25T02:00:38.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.770+0000 7f2723fff640 1 --2- 172.21.15.83:0/1935714113 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2724147350 0x7f2724140940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f271c009e80 tx=0x7f271c02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d523c14fe8453c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 1 -- 172.21.15.83:0/1935714113 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c02caf0 con 0x7f2724147350 2024-03-25T02:00:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:38.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: dump: 2024-03-25T02:00:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.781 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.782 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 1 -- 172.21.15.83:0/1935714113 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f271c02cc90 con 0x7f2724147350 2024-03-25T02:00:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 1 -- 172.21.15.83:0/1935714113 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c034610 con 0x7f2724147350 2024-03-25T02:00:38.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f27227fc640 10 monclient: dump: 2024-03-25T02:00:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.784 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f272a124640 5 monclient: authenticate success, global_id 4295 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f272a124640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.776+0000 7f272a124640 4 set_mon_vals no callback set 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.777+0000 7f272a124640 10 monclient: shutdown 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.777+0000 7f272a124640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.777+0000 7f272a124640 1 -- 172.21.15.83:0/1935714113 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2724147350 msgr2=0x7f2724140940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.777+0000 7f272a124640 1 --2- 172.21.15.83:0/1935714113 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2724147350 0x7f2724140940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f271c009e80 tx=0x7f271c02bf50 comp rx=0 tx=0).stop 2024-03-25T02:00:38.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.779+0000 7f272a124640 1 -- 172.21.15.83:0/1935714113 shutdown_connections 2024-03-25T02:00:38.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.779+0000 7f272a124640 1 --2- 172.21.15.83:0/1935714113 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2724147350 0x7f2724140940 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.789 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.779+0000 7f272a124640 1 --2- 172.21.15.83:0/1935714113 >> v2:172.21.15.83:3300/0 conn(0x7f27240b2df0 0x7f2724140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.779+0000 7f272a124640 1 -- 172.21.15.83:0/1935714113 >> 172.21.15.83:0/1935714113 conn(0x7f27240b8a00 msgr2=0x7f27240b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.779+0000 7f272a124640 1 -- 172.21.15.83:0/1935714113 shutdown_connections 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.779+0000 7f272a124640 1 -- 172.21.15.83:0/1935714113 wait complete. 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.780+0000 7f272a124640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.780+0000 7f272a124640 10 monclient: monmap: 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:38.790 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.780+0000 7f272a124640 1 librados: starting msgr at 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.780+0000 7f272a124640 1 librados: starting objecter 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.780+0000 7f272a124640 1 Processor -- start 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.781+0000 7f272a124640 1 -- start start 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.781+0000 7f272a124640 1 librados: setting wanted keys 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.781+0000 7f272a124640 1 librados: calling monclient init 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.781+0000 7f272a124640 10 monclient: init 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27240b04a0 0x7f2724145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 10 monclient(hunting): picked mon.noname-c con 0x7f27240b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240b2df0 0x7f27241d90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 10 monclient(hunting): picked mon.noname-b con 0x7f27240b2df0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:38.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2724147350 0x7f27241d9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 10 monclient(hunting): picked mon.noname-a con 0x7f2724147350 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.782+0000 7f272a124640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f272a124640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27240b0a00 con 0x7f2724147350 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f272a124640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f27237fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27240b04a0 0x7f2724145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f272a124640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27240b0880 con 0x7f27240b2df0 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f272a124640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f272a124640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f272414daf0 con 0x7f27240b04a0 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f272a124640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f272a124640 10 monclient(hunting): authenticate will time out at 1357.973495s 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f27237fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27240b04a0 0x7f2724145690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54680/0 (socket says 172.21.15.83:54680) 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f27237fe640 1 -- 172.21.15.83:0/3788208384 learned_addr learned my addr 172.21.15.83:0/3788208384 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f27237fe640 10 monclient(hunting): get_auth_request con 0x7f27240b04a0 auth_method 0 2024-03-25T02:00:38.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f27237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.783+0000 7f27237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.784+0000 7f27237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.784+0000 7f6702939640 10 monclient: get_monmap_and_config 2024-03-25T02:00:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.784+0000 7f6702939640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.784+0000 7f6702939640 10 monclient: monmap: 2024-03-25T02:00:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.793 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:38.786345+0000 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:38.786345+0000 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:38.794 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.785+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.785+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.785+0000 7f27237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.785+0000 7f6702939640 1 Processor -- start 2024-03-25T02:00:38.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.786+0000 7f6702939640 1 -- start start 2024-03-25T02:00:38.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.786+0000 7f6702939640 10 monclient: init 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.786+0000 7f6702939640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.786+0000 7f6702939640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66fc0b04a0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 10 monclient(hunting): picked mon.noname-b con 0x7f66fc0b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66fc147350 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66fc0b2410 con 0x7f66fc147350 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66fc0b26f0 con 0x7f66fc0b04a0 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f66fc0b29d0 con 0x7f66fc0b2df0 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.787+0000 7f6702939640 10 monclient(hunting): authenticate will time out at 1357.977321s 2024-03-25T02:00:38.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2024-03-25T02:00:38.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 1 -- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240b2df0 msgr2=0x7f27241d90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240b2df0 0x7f27241d90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f66fb7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 1 -- 172.21.15.83:0/3788208384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2724147350 msgr2=0x7f27241d9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2724147350 0x7f27241d9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 1 monclient: found mon.noname-c 2024-03-25T02:00:38.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.799 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.800 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.790+0000 7f27237fe640 1 -- 172.21.15.83:0/3788208384 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27241413e0 con 0x7f27240b04a0 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.792+0000 7f66fb7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc140400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54696/0 (socket says 172.21.15.83:54696) 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.792+0000 7f66fb7fe640 1 -- 172.21.15.83:0/2668522669 learned_addr learned my addr 172.21.15.83:0/2668522669 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.792+0000 7f66fb7fe640 10 monclient(hunting): get_auth_request con 0x7f66fc0b2df0 auth_method 0 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.792+0000 7f66fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.792+0000 7f66fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.792+0000 7f66fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.794+0000 7f66fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.794+0000 7f66fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.801 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.794+0000 7f66fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 1 -- 172.21.15.83:0/2668522669 >> v1:172.21.15.153:6789/0 conn(0x7f66fc0b04a0 legacy=0x7f66fc146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 1 -- 172.21.15.83:0/2668522669 >> v1:172.21.15.83:6789/0 conn(0x7f66fc147350 legacy=0x7f66fc14daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 1 monclient: found mon.noname-b 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.802 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.797+0000 7f66fb7fe640 1 -- 172.21.15.83:0/2668522669 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66fc0b3380 con 0x7f66fc0b2df0 2024-03-25T02:00:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.799+0000 7f27237fe640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27240b04a0 0x7f2724145690 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f271400c410 tx=0x7f271400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.799+0000 7f270bfff640 10 client.?.objecter ms_handle_connect 0x7f27240b04a0 2024-03-25T02:00:38.803 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.799+0000 7f270bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.805+0000 7fa3a61cf640 10 monclient: get_monmap_and_config 2024-03-25T02:00:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.805+0000 7fa3a61cf640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.805+0000 7fa3a61cf640 10 monclient: monmap: 2024-03-25T02:00:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.807 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:38.806874+0000 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:38.806874+0000 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.805+0000 7f66fb7fe640 1 --2- 172.21.15.83:0/2668522669 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc140400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098a0 tx=0x7f66f002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=490438e6c457b79f server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.806+0000 7fa3a61cf640 1 Processor -- start 2024-03-25T02:00:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.806+0000 7fa3a61cf640 1 -- start start 2024-03-25T02:00:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.806+0000 7fa3a61cf640 10 monclient: init 2024-03-25T02:00:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.806+0000 7fa3a61cf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.806+0000 7fa3a61cf640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-03-25T02:00:38.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 1 -- 172.21.15.83:0/3788208384 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714017070 con 0x7f27240b04a0 2024-03-25T02:00:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.806+0000 7fa3a61cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3a00b04a0 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:38.810 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: dump: 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f272a124640 5 monclient: authenticate success, global_id 4298 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 1 -- 172.21.15.83:0/3788208384 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27140035f0 con 0x7f27240b04a0 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3a00b2df0 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa3a0147350 0x7fa3a014eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a0147350 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a00b2410 con 0x7fa3a00b2df0 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a00b26f0 con 0x7fa3a00b04a0 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f272a124640 10 monclient: _renew_subs 2024-03-25T02:00:38.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a00b29d0 con 0x7fa3a0147350 2024-03-25T02:00:38.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.813 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7fa3a61cf640 10 monclient(hunting): authenticate will time out at 1357.997146s 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f272a124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27241416c0 con 0x7f27240b04a0 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 1 -- 172.21.15.83:0/3788208384 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714017070 con 0x7f27240b04a0 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f270bfff640 10 monclient: dump: 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.814 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.807+0000 7f2721ffb640 4 set_mon_vals no callback set 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.808+0000 7f272a124640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.808+0000 7f272a124640 10 monclient: _renew_subs 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.808+0000 7f272a124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.808+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2724141000 con 0x7f27240b04a0 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.808+0000 7f272a124640 1 librados: init done 2024-03-25T02:00:38.815 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.809+0000 7f272a124640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.809+0000 7f272a124640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.809+0000 7f272a124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.816 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.809+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2724075810 con 0x7f27240b04a0 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 1 -- 172.21.15.83:0/2668522669 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0034070 con 0x7f66fc0b2df0 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: dump: 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.818 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 1 -- 172.21.15.83:0/2668522669 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f0030840 con 0x7f66fc0b2df0 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f6702939640 5 monclient: authenticate success, global_id 4301 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f6702939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f6702939640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 1 -- 172.21.15.83:0/2668522669 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0030c10 con 0x7f66fc0b2df0 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f6702939640 4 set_mon_vals no callback set 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:38.819 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.812+0000 7f66faffd640 10 monclient: dump: 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 10 monclient: shutdown 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 1 -- 172.21.15.83:0/2668522669 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 msgr2=0x7f66fc140400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 1 --2- 172.21.15.83:0/2668522669 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc140400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f00098a0 tx=0x7f66f002b980 comp rx=0 tx=0).stop 2024-03-25T02:00:38.820 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 1 -- 172.21.15.83:0/2668522669 shutdown_connections 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 1 --2- 172.21.15.83:0/2668522669 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc140400 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 1 -- 172.21.15.83:0/2668522669 >> 172.21.15.83:0/2668522669 conn(0x7f66fc0b8a00 msgr2=0x7f66fc0b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 1 -- 172.21.15.83:0/2668522669 shutdown_connections 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.813+0000 7f6702939640 1 -- 172.21.15.83:0/2668522669 wait complete. 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.814+0000 7f6702939640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.814+0000 7f6702939640 10 monclient: monmap: 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:38.821 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.815+0000 7f6702939640 1 librados: starting msgr at 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.815+0000 7f6702939640 1 librados: starting objecter 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.815+0000 7f6702939640 1 Processor -- start 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.815+0000 7f6702939640 1 -- start start 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.815+0000 7f6702939640 1 librados: setting wanted keys 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.815+0000 7f6702939640 1 librados: calling monclient init 2024-03-25T02:00:38.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.815+0000 7f6702939640 10 monclient: init 2024-03-25T02:00:38.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.816+0000 7f6702939640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.824 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.816+0000 7f6702939640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.816+0000 7f6702939640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66fc0b04a0 0x7f66fc1cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.816+0000 7f6702939640 10 monclient(hunting): picked mon.noname-a con 0x7f66fc0b04a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.816+0000 7f6702939640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc138d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 10 monclient(hunting): picked mon.noname-c con 0x7f66fc0b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66fc147350 0x7f66fc139250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 10 monclient(hunting): picked mon.noname-b con 0x7f66fc147350 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc140ac0 con 0x7f66fc0b04a0 2024-03-25T02:00:38.825 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc140940 con 0x7f66fc147350 2024-03-25T02:00:38.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f66fc140c40 con 0x7f66fc0b2df0 2024-03-25T02:00:38.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.817+0000 7f6702939640 10 monclient(hunting): authenticate will time out at 1358.007282s 2024-03-25T02:00:38.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.831+0000 7fa434414640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-25T02:00:38.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.831+0000 7fa434414640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-25T02:00:38.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.831+0000 7fa434414640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.831+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa42c06cbd0 con 0x7fa42c06daf0 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.854+0000 7f20fe7fc640 1 -- 172.21.15.83:0/121833758 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f21000574a0 con 0x7f21080b33f0 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.855+0000 7f20fe7fc640 10 client.4365.objecter ms_dispatch 0x7f2108000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.855+0000 7f20fe7fc640 3 client.4365.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.855+0000 7f20fe7fc640 3 client.4365.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.855+0000 7f20fe7fc640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.855+0000 7f20fe7fc640 1 -- 172.21.15.83:0/121833758 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2100023020 con 0x7f21080b33f0 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.855+0000 7f20fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.855+0000 7f20fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:38.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.863+0000 7fa39ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa3a0147350 0x7fa3a014eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.864+0000 7fa39ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fa3a0147350 0x7fa3a014eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56200/0 (socket says 172.21.15.83:56200) 2024-03-25T02:00:38.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.864+0000 7fa39ffff640 1 -- 172.21.15.83:0/1609443968 learned_addr learned my addr 172.21.15.83:0/1609443968 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.864+0000 7fa39ffff640 10 monclient(hunting): get_auth_request con 0x7fa3a0147350 auth_method 0 2024-03-25T02:00:38.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.864+0000 7fa39ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.864+0000 7fa39ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.864+0000 7fa39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.868+0000 7fa39ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.868+0000 7fa39ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.868+0000 7fa39ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2024-03-25T02:00:38.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 1 -- 172.21.15.83:0/1609443968 >> v1:172.21.15.153:6789/0 conn(0x7fa3a00b04a0 legacy=0x7fa3a0146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 1 -- 172.21.15.83:0/1609443968 >> v1:172.21.15.83:6789/0 conn(0x7fa3a00b2df0 legacy=0x7fa3a0140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:38.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 1 monclient: found mon.noname-a 2024-03-25T02:00:38.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.870+0000 7fa39ffff640 1 -- 172.21.15.83:0/1609443968 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a00b3380 con 0x7fa3a0147350 2024-03-25T02:00:38.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.875+0000 7f67957fa640 1 -- 172.21.15.83:0/2636188064 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f678c021020 con 0x7f6798070e80 2024-03-25T02:00:38.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.875+0000 7f67957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.875+0000 7f67957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:38.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.880+0000 7f6700ed6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66fc147350 0x7f66fc139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.882+0000 7f6700ed6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66fc147350 0x7f66fc139250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40806/0 (socket says 172.21.15.83:40806) 2024-03-25T02:00:38.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.882+0000 7f6700ed6640 1 -- 172.21.15.83:0/1252593630 learned_addr learned my addr 172.21.15.83:0/1252593630 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.882+0000 7f6700ed6640 10 monclient(hunting): get_auth_request con 0x7f66fc147350 auth_method 0 2024-03-25T02:00:38.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.882+0000 7f6700ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.882+0000 7f6700ed6640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.882+0000 7f6700ed6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.884+0000 7f6700ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.884+0000 7f6700ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.884+0000 7f6700ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.890 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-03-25T02:00:38.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 1 -- 172.21.15.83:0/1252593630 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 msgr2=0x7f66fc138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc138d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 1 -- 172.21.15.83:0/1252593630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66fc0b04a0 msgr2=0x7f66fc1cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66fc0b04a0 0x7f66fc1cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 1 monclient: found mon.noname-b 2024-03-25T02:00:38.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.888+0000 7f6700ed6640 1 -- 172.21.15.83:0/1252593630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66fc1ce3d0 con 0x7f66fc147350 2024-03-25T02:00:38.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.891+0000 7f6700ed6640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66fc147350 0x7f66fc139250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66f400b990 tx=0x7f66f400be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.891+0000 7f66f8ff9640 10 client.?.objecter ms_handle_connect 0x7f66fc147350 2024-03-25T02:00:38.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.891+0000 7f66f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:38.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.894+0000 7fa422ffd640 1 -- 172.21.15.83:0/59789742 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa42c072210 con 0x7fa42c06daf0 2024-03-25T02:00:38.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.894+0000 7fa422ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-03-25T02:00:38.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.894+0000 7fa422ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:38.897 INFO:teuthology.orchestra.run.smithi083.stdout:85899345930 2024-03-25T02:00:38.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 1 -- 172.21.15.83:0/1252593630 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4019070 con 0x7f66fc147350 2024-03-25T02:00:38.902 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:38.903 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: dump: 2024-03-25T02:00:38.903 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.903 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.903 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.903 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.904 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.904 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.904 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.904 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.904 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.905 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 1 -- 172.21.15.83:0/1252593630 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f400cba0 con 0x7f66fc147350 2024-03-25T02:00:38.905 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 1 -- 172.21.15.83:0/1252593630 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f4014440 con 0x7f66fc147350 2024-03-25T02:00:38.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:38.906 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66f8ff9640 10 monclient: dump: 2024-03-25T02:00:38.906 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.906 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.907 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.907 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.907 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.907 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.907 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.907 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.908 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.908 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f66fa7fc640 4 set_mon_vals no callback set 2024-03-25T02:00:38.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.895+0000 7f6702939640 5 monclient: authenticate success, global_id 4408 2024-03-25T02:00:38.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 10 monclient: _renew_subs 2024-03-25T02:00:38.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66fc1ccd30 con 0x7f66fc147350 2024-03-25T02:00:38.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:38.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 10 monclient: _renew_subs 2024-03-25T02:00:38.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66fc139790 con 0x7f66fc147350 2024-03-25T02:00:38.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.896+0000 7f6702939640 1 librados: init done 2024-03-25T02:00:38.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7f6702939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7f6702939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7f6702939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:38.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66fc075810 con 0x7f66fc147350 2024-03-25T02:00:38.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 10 librados: watch_flush enter 2024-03-25T02:00:38.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 10 librados: watch_flush exit 2024-03-25T02:00:38.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T02:00:38.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 10 client.4396.objecter successfully canceled tick 2024-03-25T02:00:38.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa41006da70 msgr2=0x7fa41006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa41006da70 0x7fa41006ff30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa41c036000 tx=0x7fa41c030000 comp rx=0 tx=0).stop 2024-03-25T02:00:38.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 10 monclient: shutdown 2024-03-25T02:00:38.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.897+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c06daf0 msgr2=0x7fa42c0adbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c06daf0 0x7fa42c0adbe0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa42800c370 tx=0x7fa42800c830 comp rx=0 tx=0).stop 2024-03-25T02:00:38.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 shutdown_connections 2024-03-25T02:00:38.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa41006da70 0x7fa41006ff30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa42c1406d0 0x7fa42c0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c072660 0x7fa42c1d87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 --2- 172.21.15.83:0/59789742 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa42c06daf0 0x7fa42c0adbe0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 >> 172.21.15.83:0/59789742 conn(0x7fa42c067e90 msgr2=0x7fa42c13f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 shutdown_connections 2024-03-25T02:00:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 -- 172.21.15.83:0/59789742 wait complete. 2024-03-25T02:00:38.916 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.898+0000 7fa434414640 1 librados: shutdown 2024-03-25T02:00:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.899+0000 7fa39ffff640 1 --2- 172.21.15.83:0/1609443968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a0147350 0x7fa3a014eb00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa398009e80 tx=0x7fa39802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629d8e2ea14e9257 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.909+0000 7f6b948c5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-25T02:00:38.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.909+0000 7f6b948c5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-25T02:00:38.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.909+0000 7f6b948c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:38.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.909+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f6b90075810 con 0x7f6b90140400 2024-03-25T02:00:38.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345930 for osd.2 2024-03-25T02:00:38.920 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:38.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.915+0000 7f66f8ff9640 1 -- 172.21.15.83:0/1252593630 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f66f401c070 con 0x7f66fc147350 2024-03-25T02:00:38.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.916+0000 7f66f8ff9640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f66e806da70 0x7f66e806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.916+0000 7f66fbfff640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f66e806da70 0x7f66e806ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.916+0000 7f66fbfff640 10 monclient: get_auth_request con 0x7f66e806da70 auth_method 0 2024-03-25T02:00:38.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.916+0000 7f66f8ff9640 10 client.4408.objecter ms_handle_connect 0x7f66e806da70 2024-03-25T02:00:38.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.916+0000 7f66fbfff640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f66e806da70 0x7f66e806ff30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f66ec001560 tx=0x7f66ec007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 1 -- 172.21.15.83:0/1609443968 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39802ca70 con 0x7fa3a0147350 2024-03-25T02:00:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: dump: 2024-03-25T02:00:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.927 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.928 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.929 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.929 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 1 -- 172.21.15.83:0/1609443968 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa39802cc10 con 0x7fa3a0147350 2024-03-25T02:00:38.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 1 -- 172.21.15.83:0/1609443968 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3980343f0 con 0x7fa3a0147350 2024-03-25T02:00:38.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 5 monclient: authenticate success, global_id 4368 2024-03-25T02:00:38.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:38.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 4 set_mon_vals no callback set 2024-03-25T02:00:38.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:38.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa39e7fc640 10 monclient: dump: 2024-03-25T02:00:38.931 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.931 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.931 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.931 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.932 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.933 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 10 monclient: shutdown 2024-03-25T02:00:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 1 -- 172.21.15.83:0/1609443968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a0147350 msgr2=0x7fa3a014eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/1609443968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a0147350 0x7fa3a014eb00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa398009e80 tx=0x7fa39802bed0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 1 -- 172.21.15.83:0/1609443968 shutdown_connections 2024-03-25T02:00:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.919+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/1609443968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a0147350 0x7fa3a014eb00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.920+0000 7fa3a61cf640 1 -- 172.21.15.83:0/1609443968 >> 172.21.15.83:0/1609443968 conn(0x7fa3a00b8a00 msgr2=0x7fa3a00b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.920+0000 7fa3a61cf640 1 -- 172.21.15.83:0/1609443968 shutdown_connections 2024-03-25T02:00:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.920+0000 7fa3a61cf640 1 -- 172.21.15.83:0/1609443968 wait complete. 2024-03-25T02:00:38.934 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.920+0000 7fa3a61cf640 10 monclient: build_initial_monmap 2024-03-25T02:00:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.920+0000 7fa3a61cf640 10 monclient: monmap: 2024-03-25T02:00:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:38.935 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:38.936 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.936 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:38.936 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:38.936 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:38.936 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.921+0000 7fa3a61cf640 1 librados: starting msgr at 2024-03-25T02:00:38.936 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.921+0000 7fa3a61cf640 1 librados: starting objecter 2024-03-25T02:00:38.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.921+0000 7fa3a61cf640 1 Processor -- start 2024-03-25T02:00:38.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.921+0000 7fa3a61cf640 1 -- start start 2024-03-25T02:00:38.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.921+0000 7fa3a61cf640 1 librados: setting wanted keys 2024-03-25T02:00:38.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.921+0000 7fa3a61cf640 1 librados: calling monclient init 2024-03-25T02:00:38.937 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.921+0000 7fa3a61cf640 10 monclient: init 2024-03-25T02:00:38.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:38.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:00:38.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a00b04a0 0x7fa3a0143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa3a00b04a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:38.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3a00b2df0 0x7fa3a01dc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.938 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa3a00b2df0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:38.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa3a0147350 0x7fa3a0144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa3a0147350 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:38.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a0140c90 con 0x7fa3a00b04a0 2024-03-25T02:00:38.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.939 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a0140b10 con 0x7fa3a00b2df0 2024-03-25T02:00:38.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:38.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a0140e10 con 0x7fa3a0147350 2024-03-25T02:00:38.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:38.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa3a61cf640 10 monclient(hunting): authenticate will time out at 1358.112167s 2024-03-25T02:00:38.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.922+0000 7fa39f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a00b04a0 0x7fa3a0143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a00b04a0 0x7fa3a0143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56224/0 (socket says 172.21.15.83:56224) 2024-03-25T02:00:38.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 1 -- 172.21.15.83:0/3043023505 learned_addr learned my addr 172.21.15.83:0/3043023505 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:38.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 10 monclient(hunting): get_auth_request con 0x7fa3a00b04a0 auth_method 0 2024-03-25T02:00:38.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:38.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:38.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:38.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:38.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:38.942 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.923+0000 7fa39f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:38.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-03-25T02:00:38.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 1 -- 172.21.15.83:0/3043023505 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa3a0147350 msgr2=0x7fa3a0144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa3a0147350 0x7fa3a0144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 1 -- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3a00b2df0 msgr2=0x7fa3a01dc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:38.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3a00b2df0 0x7fa3a01dc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 1 monclient: found mon.noname-a 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.926+0000 7fa39f7fe640 1 -- 172.21.15.83:0/3043023505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a01433b0 con 0x7fa3a00b04a0 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.928+0000 7fa39f7fe640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a00b04a0 0x7fa3a0143ab0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa39000c440 tx=0x7fa39000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.928+0000 7fa387fff640 10 client.?.objecter ms_handle_connect 0x7fa3a00b04a0 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.928+0000 7fa387fff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 1 -- 172.21.15.83:0/3043023505 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390017070 con 0x7fa3a00b04a0 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: dump: 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.947 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: _finish_auth 0 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 1 -- 172.21.15.83:0/3043023505 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3900035f0 con 0x7fa3a00b04a0 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa3a61cf640 5 monclient: authenticate success, global_id 4371 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa3a61cf640 10 monclient: _renew_subs 2024-03-25T02:00:38.948 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa3a61cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a0143690 con 0x7fa3a00b04a0 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 1 -- 172.21.15.83:0/3043023505 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3900039e0 con 0x7fa3a00b04a0 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa387fff640 10 monclient: dump: 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.932+0000 7fa39dffb640 4 set_mon_vals no callback set 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.933+0000 7fa3a61cf640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.933+0000 7fa3a61cf640 10 monclient: _renew_subs 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.933+0000 7fa3a61cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.933+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a01dcf60 con 0x7fa3a00b04a0 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.933+0000 7fa3a61cf640 1 librados: init done 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.934+0000 7fa3a61cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.934+0000 7fa3a61cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.934+0000 7fa3a61cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:38.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.934+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a0075810 con 0x7fa3a00b04a0 2024-03-25T02:00:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.935+0000 7f270bfff640 1 -- 172.21.15.83:0/3788208384 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f2714007020 con 0x7f27240b04a0 2024-03-25T02:00:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.936+0000 7f270bfff640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f271006dc20 0x7f27100700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.936+0000 7f2722ffd640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f271006dc20 0x7f27100700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.936+0000 7f2722ffd640 10 monclient: get_auth_request con 0x7f271006dc20 auth_method 0 2024-03-25T02:00:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.937+0000 7f2722ffd640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f271006dc20 0x7f27100700e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f27240b6a20 tx=0x7f2718006c90 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.937+0000 7f270bfff640 10 client.4298.objecter ms_handle_connect 0x7f271006dc20 2024-03-25T02:00:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.958+0000 7fa387fff640 1 -- 172.21.15.83:0/3043023505 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fa390003b80 con 0x7fa3a00b04a0 2024-03-25T02:00:38.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.959+0000 7fa387fff640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa38c06da20 0x7fa38c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.959+0000 7fa39effd640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa38c06da20 0x7fa38c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.959+0000 7fa39effd640 10 monclient: get_auth_request con 0x7fa38c06da20 auth_method 0 2024-03-25T02:00:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.959+0000 7fa387fff640 10 client.4371.objecter ms_handle_connect 0x7fa38c06da20 2024-03-25T02:00:38.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.959+0000 7fa39effd640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa38c06da20 0x7fa38c06fee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3a01462d0 tx=0x7fa394007990 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:38.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.962+0000 7fa387fff640 1 -- 172.21.15.83:0/3043023505 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fa3900566f0 con 0x7fa3a00b04a0 2024-03-25T02:00:38.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.962+0000 7fa387fff640 10 client.4371.objecter ms_dispatch 0x7fa3a0000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.962+0000 7fa387fff640 3 client.4371.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.962+0000 7fa387fff640 3 client.4371.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:38.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:38.962+0000 7fa387fff640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T02:00:39.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.083+0000 7f210fd79640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:39.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.083+0000 7f210fd79640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:39.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.083+0000 7f210fd79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:39.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.083+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f21080aa9a0 con 0x7f21080b33f0 2024-03-25T02:00:39.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.087+0000 7f679ec08640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:39.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.087+0000 7f679ec08640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:39.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.087+0000 7f679ec08640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:39.089 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.087+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f67981dbf60 con 0x7f6798070e80 2024-03-25T02:00:39.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 1 -- 172.21.15.83:0/3788208384 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f27140562f0 con 0x7f27240b04a0 2024-03-25T02:00:39.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 10 client.4298.objecter ms_dispatch 0x7f2724000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:39.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 3 client.4298.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:39.124 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 3 client.4298.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:39.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T02:00:39.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 1 -- 172.21.15.83:0/3788208384 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2714058950 con 0x7f27240b04a0 2024-03-25T02:00:39.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:39.125 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.122+0000 7f270bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:39.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.136+0000 7f6b7effd640 1 -- 172.21.15.83:0/1738840272 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6b80021020 con 0x7f6b90140400 2024-03-25T02:00:39.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.136+0000 7f6b7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-03-25T02:00:39.138 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.136+0000 7f6b7effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:39.139 INFO:teuthology.orchestra.run.smithi083.stdout:64424509447 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 10 librados: watch_flush enter 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 10 librados: watch_flush exit 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 10 client.4292.objecter successfully canceled tick 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6b7006da20 msgr2=0x7f6b7006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6b7006da20 0x7f6b7006fee0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6b88035000 tx=0x7f6b88030000 comp rx=0 tx=0).stop 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 10 monclient: shutdown 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b90140400 msgr2=0x7f6b901dc240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.141 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b90140400 0x7f6b901dc240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000c3c0 tx=0x7f6b8000c880 comp rx=0 tx=0).stop 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 shutdown_connections 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f6b7006da20 0x7f6b7006fee0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6b90140400 0x7f6b901dc240 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b900b2df0 0x7f6b901dbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 --2- 172.21.15.83:0/1738840272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b900b04a0 0x7f6b90143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.138+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 >> 172.21.15.83:0/1738840272 conn(0x7f6b900b8a00 msgr2=0x7f6b900b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.139+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 shutdown_connections 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.139+0000 7f6b948c5640 1 -- 172.21.15.83:0/1738840272 wait complete. 2024-03-25T02:00:39.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.139+0000 7f6b948c5640 1 librados: shutdown 2024-03-25T02:00:39.152 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.0 2024-03-25T02:00:39.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.150+0000 7f66f8ff9640 1 -- 172.21.15.83:0/1252593630 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f66f40574e0 con 0x7f66fc147350 2024-03-25T02:00:39.152 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.150+0000 7f66f8ff9640 10 client.4408.objecter ms_dispatch 0x7f66fc000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:39.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.150+0000 7f66f8ff9640 3 client.4408.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:39.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.150+0000 7f66f8ff9640 3 client.4408.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:39.153 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.150+0000 7f66f8ff9640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T02:00:39.153 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.163+0000 7f66f8ff9640 1 -- 172.21.15.83:0/1252593630 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f66f4023020 con 0x7f66fc147350 2024-03-25T02:00:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.163+0000 7f66f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:39.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.163+0000 7f66f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.164+0000 7f67957fa640 1 -- 172.21.15.83:0/2636188064 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f678c059220 con 0x7f6798070e80 2024-03-25T02:00:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.164+0000 7f67957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:39.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.164+0000 7f67957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:39.166 INFO:teuthology.orchestra.run.smithi083.stdout:60129542152 2024-03-25T02:00:39.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 10 librados: watch_flush enter 2024-03-25T02:00:39.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 10 librados: watch_flush exit 2024-03-25T02:00:39.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 10 client.4405.objecter successfully canceled tick 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677c06da70 msgr2=0x7f677c06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677c06da70 0x7f677c06ff30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f679813bf60 tx=0x7f6784007990 comp rx=0 tx=0).stop 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 10 monclient: shutdown 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798070e80 msgr2=0x7f67980adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798070e80 0x7f67980adb60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f678c00c530 tx=0x7f678c00c9f0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 shutdown_connections 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677c06da70 0x7f677c06ff30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6798148e50 0x7f67981408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6798072e20 0x7f67981d8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 --2- 172.21.15.83:0/2636188064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6798070e80 0x7f67980adb60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 >> 172.21.15.83:0/2636188064 conn(0x7f67980b4a10 msgr2=0x7f6798138740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:39.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 shutdown_connections 2024-03-25T02:00:39.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 -- 172.21.15.83:0/2636188064 wait complete. 2024-03-25T02:00:39.171 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.167+0000 7f679ec08640 1 librados: shutdown 2024-03-25T02:00:39.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.4 2024-03-25T02:00:39.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.229+0000 7fa387fff640 1 -- 172.21.15.83:0/3043023505 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa390058d50 con 0x7fa3a00b04a0 2024-03-25T02:00:39.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.230+0000 7fa387fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:39.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.230+0000 7fa387fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.245+0000 7f20fe7fc640 1 -- 172.21.15.83:0/121833758 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2100059b00 con 0x7f21080b33f0 2024-03-25T02:00:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.245+0000 7f20fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:39.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.245+0000 7f20fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:39.247 INFO:teuthology.orchestra.run.smithi083.stdout:81604378631 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 10 librados: watch_flush enter 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 10 librados: watch_flush exit 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 10 client.4365.objecter successfully canceled tick 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f20ec06da20 msgr2=0x7f20ec06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f20ec06da20 0x7f20ec06fee0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2104036000 tx=0x7f2104030000 comp rx=0 tx=0).stop 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 10 monclient: shutdown 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080b33f0 msgr2=0x7f21081cd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080b33f0 0x7f21081cd320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f210000b9e0 tx=0x7f210000bea0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 shutdown_connections 2024-03-25T02:00:39.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f20ec06da20 0x7f20ec06fee0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f21080b33f0 0x7f21081cd320 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f21080a9f50 0x7f21081ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 --2- 172.21.15.83:0/121833758 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f21080a9b70 0x7f21081cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 >> 172.21.15.83:0/121833758 conn(0x7f2108067e30 msgr2=0x7f210813e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:39.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 shutdown_connections 2024-03-25T02:00:39.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 -- 172.21.15.83:0/121833758 wait complete. 2024-03-25T02:00:39.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.248+0000 7f210fd79640 1 librados: shutdown 2024-03-25T02:00:39.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378632 got 81604378631 for osd.7 2024-03-25T02:00:39.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.363+0000 7f6702939640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-25T02:00:39.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.363+0000 7f6702939640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-25T02:00:39.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.363+0000 7f6702939640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:39.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.363+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f66fc073910 con 0x7f66fc147350 2024-03-25T02:00:39.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.366+0000 7f66f8ff9640 1 -- 172.21.15.83:0/1252593630 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f66f4059b40 con 0x7f66fc147350 2024-03-25T02:00:39.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.366+0000 7f66f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-03-25T02:00:39.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.366+0000 7f66f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:39.369 INFO:teuthology.orchestra.run.smithi083.stdout:64424509447 2024-03-25T02:00:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 10 librados: watch_flush enter 2024-03-25T02:00:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 10 librados: watch_flush exit 2024-03-25T02:00:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T02:00:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f272a124640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-25T02:00:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f272a124640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-25T02:00:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f272a124640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:39.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 10 client.4408.objecter successfully canceled tick 2024-03-25T02:00:39.372 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f27240b73d0 con 0x7f27240b04a0 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f66e806da70 msgr2=0x7f66e806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f66e806da70 0x7f66e806ff30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f66ec001560 tx=0x7f66ec007990 comp rx=0 tx=0).stop 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 10 monclient: shutdown 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66fc147350 msgr2=0x7f66fc139250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66fc147350 0x7f66fc139250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f66f400b990 tx=0x7f66f400be50 comp rx=0 tx=0).stop 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 shutdown_connections 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f66e806da70 0x7f66e806ff30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f66fc147350 0x7f66fc139250 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f66fc0b2df0 0x7f66fc138d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 --2- 172.21.15.83:0/1252593630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66fc0b04a0 0x7f66fc1cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 >> 172.21.15.83:0/1252593630 conn(0x7f66fc0b8a00 msgr2=0x7f66fc0b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:39.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 shutdown_connections 2024-03-25T02:00:39.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 -- 172.21.15.83:0/1252593630 wait complete. 2024-03-25T02:00:39.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.369+0000 7f6702939640 1 librados: shutdown 2024-03-25T02:00:39.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.375+0000 7f270bfff640 1 -- 172.21.15.83:0/3788208384 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2714058250 con 0x7f27240b04a0 2024-03-25T02:00:39.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.375+0000 7f270bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-03-25T02:00:39.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.375+0000 7f270bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:39.378 INFO:teuthology.orchestra.run.smithi083.stdout:81604378634 2024-03-25T02:00:39.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.377+0000 7f272a124640 10 librados: watch_flush enter 2024-03-25T02:00:39.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 10 librados: watch_flush exit 2024-03-25T02:00:39.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T02:00:39.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 10 client.4298.objecter successfully canceled tick 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f271006dc20 msgr2=0x7f27100700e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f271006dc20 0x7f27100700e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f27240b6a20 tx=0x7f2718006c90 comp rx=0 tx=0).stop 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 10 monclient: shutdown 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27240b04a0 msgr2=0x7f2724145690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27240b04a0 0x7f2724145690 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f271400c410 tx=0x7f271400c8d0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 shutdown_connections 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f271006dc20 0x7f27100700e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2724147350 0x7f27241d9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27240b2df0 0x7f27241d90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 --2- 172.21.15.83:0/3788208384 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f27240b04a0 0x7f2724145690 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 >> 172.21.15.83:0/3788208384 conn(0x7f27240b8a00 msgr2=0x7f272413f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 shutdown_connections 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 -- 172.21.15.83:0/3788208384 wait complete. 2024-03-25T02:00:39.381 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.378+0000 7f272a124640 1 librados: shutdown 2024-03-25T02:00:39.385 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.3 2024-03-25T02:00:39.385 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:39.392 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378634 got 81604378634 for osd.6 2024-03-25T02:00:39.392 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.451+0000 7fa3a61cf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-25T02:00:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.451+0000 7fa3a61cf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-25T02:00:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.451+0000 7fa3a61cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.451+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa3a00b7740 con 0x7fa3a00b04a0 2024-03-25T02:00:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.455+0000 7fa387fff640 1 -- 172.21.15.83:0/3043023505 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa3900010c0 con 0x7fa3a00b04a0 2024-03-25T02:00:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.455+0000 7fa387fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-03-25T02:00:39.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.455+0000 7fa387fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:39.458 INFO:teuthology.orchestra.run.smithi083.stdout:77309411335 2024-03-25T02:00:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.457+0000 7fa3a61cf640 10 librados: watch_flush enter 2024-03-25T02:00:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 10 librados: watch_flush exit 2024-03-25T02:00:39.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 10 client.4371.objecter successfully canceled tick 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa38c06da20 msgr2=0x7fa38c06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa38c06da20 0x7fa38c06fee0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa3a01462d0 tx=0x7fa394007990 comp rx=0 tx=0).stop 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 10 monclient: shutdown 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a00b04a0 msgr2=0x7fa3a0143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a00b04a0 0x7fa3a0143ab0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa39000c440 tx=0x7fa39000c900 comp rx=0 tx=0).stop 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 shutdown_connections 2024-03-25T02:00:39.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa38c06da20 0x7fa38c06fee0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa3a0147350 0x7fa3a0144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa3a00b2df0 0x7fa3a01dc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 --2- 172.21.15.83:0/3043023505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa3a00b04a0 0x7fa3a0143ab0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:39.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 >> 172.21.15.83:0/3043023505 conn(0x7fa3a00b8a00 msgr2=0x7fa3a00b8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:39.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 shutdown_connections 2024-03-25T02:00:39.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 -- 172.21.15.83:0/3043023505 wait complete. 2024-03-25T02:00:39.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:39.458+0000 7fa3a61cf640 1 librados: shutdown 2024-03-25T02:00:39.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411334 got 77309411335 for osd.5 2024-03-25T02:00:39.471 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:40.182 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-25T02:00:40.267 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-25T02:00:40.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.288+0000 7fbdda904640 10 monclient: get_monmap_and_config 2024-03-25T02:00:40.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.288+0000 7fbdda904640 10 monclient: build_initial_monmap 2024-03-25T02:00:40.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.288+0000 7fbdda904640 10 monclient: monmap: 2024-03-25T02:00:40.290 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:40.290 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:40.289717+0000 2024-03-25T02:00:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:40.289717+0000 2024-03-25T02:00:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:40.291 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:40.292 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:40.292 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:40.292 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:40.292 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 1 Processor -- start 2024-03-25T02:00:40.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 1 -- start start 2024-03-25T02:00:40.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient: init 2024-03-25T02:00:40.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:40.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-03-25T02:00:40.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbdd40b04a0 0x7fbdd4146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd40b04a0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:40.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b2df0 0x7fbdd4147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd40b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:40.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbdd4140400 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:40.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdd40b2410 con 0x7fbdd4140400 2024-03-25T02:00:40.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fbdd40b26f0 con 0x7fbdd40b04a0 2024-03-25T02:00:40.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd40b29d0 con 0x7fbdd40b2df0 2024-03-25T02:00:40.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:40.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.289+0000 7fbdda904640 10 monclient(hunting): authenticate will time out at 1359.479741s 2024-03-25T02:00:40.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.293+0000 7fbdd3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbdd40b04a0 0x7fbdd4146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.296+0000 7fbdd37fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b2df0 0x7fbdd4147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.298+0000 7fbdd37fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b2df0 0x7fbdd4147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54726/0 (socket says 172.21.15.83:54726) 2024-03-25T02:00:40.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.298+0000 7fbdd37fe640 1 -- 172.21.15.83:0/427484695 learned_addr learned my addr 172.21.15.83:0/427484695 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:40.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.298+0000 7fbdd37fe640 10 monclient(hunting): get_auth_request con 0x7fbdd40b2df0 auth_method 0 2024-03-25T02:00:40.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.298+0000 7fbdd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:40.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.298+0000 7fbdd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:40.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.298+0000 7fbdd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:40.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.299+0000 7fbdd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:40.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.299+0000 7fbdd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:40.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.299+0000 7fbdd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:40.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-03-25T02:00:40.303 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 1 -- 172.21.15.83:0/427484695 >> v2:172.21.15.153:3300/0 conn(0x7fbdd40b04a0 msgr2=0x7fbdd4146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:40.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 1 --2- 172.21.15.83:0/427484695 >> v2:172.21.15.153:3300/0 conn(0x7fbdd40b04a0 0x7fbdd4146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 1 -- 172.21.15.83:0/427484695 >> v1:172.21.15.153:6789/0 conn(0x7fbdd4140400 legacy=0x7fbdd4140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:40.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 10 monclient: _finish_hunting 0 2024-03-25T02:00:40.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 1 monclient: found mon.noname-b 2024-03-25T02:00:40.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:40.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:40.304 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.301+0000 7fbdd37fe640 1 -- 172.21.15.83:0/427484695 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd40b3380 con 0x7fbdd40b2df0 2024-03-25T02:00:40.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.303+0000 7fbdd37fe640 1 --2- 172.21.15.83:0/427484695 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b2df0 0x7fbdd4147180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009870 tx=0x7fbdc802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47fd22b6d7f0bcb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 1 -- 172.21.15.83:0/427484695 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8034070 con 0x7fbdd40b2df0 2024-03-25T02:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: dump: 2024-03-25T02:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.359 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: _finish_auth 0 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:40.360 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 1 -- 172.21.15.83:0/427484695 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc8030840 con 0x7fbdd40b2df0 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdda904640 5 monclient: authenticate success, global_id 4304 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 1 -- 172.21.15.83:0/427484695 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8030c10 con 0x7fbdd40b2df0 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdd2ffd640 10 monclient: dump: 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.361 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.362 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.362 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.362 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdda904640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:40.362 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.357+0000 7fbdda904640 4 set_mon_vals no callback set 2024-03-25T02:00:40.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 10 monclient: shutdown 2024-03-25T02:00:40.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:40.365 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 -- 172.21.15.83:0/427484695 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b2df0 msgr2=0x7fbdd4147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 --2- 172.21.15.83:0/427484695 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b2df0 0x7fbdd4147180 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009870 tx=0x7fbdc802b980 comp rx=0 tx=0).stop 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 -- 172.21.15.83:0/427484695 shutdown_connections 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 --2- 172.21.15.83:0/427484695 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b2df0 0x7fbdd4147180 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 --2- 172.21.15.83:0/427484695 >> v2:172.21.15.153:3300/0 conn(0x7fbdd40b04a0 0x7fbdd4146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 -- 172.21.15.83:0/427484695 >> 172.21.15.83:0/427484695 conn(0x7fbdd40b8a00 msgr2=0x7fbdd40b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 -- 172.21.15.83:0/427484695 shutdown_connections 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.358+0000 7fbdda904640 1 -- 172.21.15.83:0/427484695 wait complete. 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.359+0000 7fbdda904640 10 monclient: build_initial_monmap 2024-03-25T02:00:40.366 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.359+0000 7fbdda904640 10 monclient: monmap: 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.359+0000 7fbdda904640 1 librados: starting msgr at 2024-03-25T02:00:40.367 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.359+0000 7fbdda904640 1 librados: starting objecter 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.359+0000 7fbdda904640 1 Processor -- start 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.359+0000 7fbdda904640 1 -- start start 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 librados: setting wanted keys 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 librados: calling monclient init 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient: init 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b04a0 0x7fbdd4130c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.368 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd40b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd40b2df0 0x7fbdd413de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd40b2df0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdd4140400 0x7fbdd4133320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd4140400 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd414dc70 con 0x7fbdd4140400 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd414daf0 con 0x7fbdd40b2df0 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fbdd414ddf0 con 0x7fbdd40b04a0 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.360+0000 7fbdda904640 10 monclient(hunting): authenticate will time out at 1359.550863s 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd3fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b04a0 0x7fbdd4130c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd8ea1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdd4140400 0x7fbdd4133320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.369 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd3fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b04a0 0x7fbdd4130c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54740/0 (socket says 172.21.15.83:54740) 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd3fff640 1 -- 172.21.15.83:0/2502438168 learned_addr learned my addr 172.21.15.83:0/2502438168 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd3fff640 10 monclient(hunting): get_auth_request con 0x7fbdd40b04a0 auth_method 0 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd8ea1640 10 monclient(hunting): get_auth_request con 0x7fbdd4140400 auth_method 0 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd8ea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd8ea1640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:40.370 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.361+0000 7fbdd8ea1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.362+0000 7fbdd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.362+0000 7fbdd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.362+0000 7fbdd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.362+0000 7fbdd8ea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.362+0000 7fbdd8ea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:40.371 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.362+0000 7fbdd8ea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 1 -- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd40b2df0 msgr2=0x7fbdd413de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd40b2df0 0x7fbdd413de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 1 -- 172.21.15.83:0/2502438168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdd4140400 msgr2=0x7fbdd4133320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdd4140400 0x7fbdd4133320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 10 monclient: _finish_hunting 0 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 1 monclient: found mon.noname-c 2024-03-25T02:00:40.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.364+0000 7fbdd3fff640 1 -- 172.21.15.83:0/2502438168 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdd4134640 con 0x7fbdd40b04a0 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.367+0000 7fbdd3fff640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b04a0 0x7fbdd4130c40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c7f0 tx=0x7fbdc400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.367+0000 7fbdd0ff9640 10 client.?.objecter ms_handle_connect 0x7fbdd40b04a0 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.367+0000 7fbdd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 1 -- 172.21.15.83:0/2502438168 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4015070 con 0x7fbdd40b04a0 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: dump: 2024-03-25T02:00:40.374 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: _finish_auth 0 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: _check_auth_tickets 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 1 -- 172.21.15.83:0/2502438168 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc4004240 con 0x7fbdd40b04a0 2024-03-25T02:00:40.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 5 monclient: authenticate success, global_id 4307 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 10 monclient: _renew_subs 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdd4134920 con 0x7fbdd40b04a0 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 1 -- 172.21.15.83:0/2502438168 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc4003610 con 0x7fbdd40b04a0 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd0ff9640 10 monclient: dump: 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.376 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdd27fc640 4 set_mon_vals no callback set 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 10 monclient: _renew_subs 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdd4133860 con 0x7fbdd40b04a0 2024-03-25T02:00:40.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.371+0000 7fbdda904640 1 librados: init done 2024-03-25T02:00:40.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.372+0000 7fbdda904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:40.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.372+0000 7fbdda904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:40.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.372+0000 7fbdda904640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:40.378 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.372+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdd4075810 con 0x7fbdd40b04a0 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.376+0000 7fa56036d640 10 monclient: get_monmap_and_config 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.376+0000 7fa56036d640 10 monclient: build_initial_monmap 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.376+0000 7fa56036d640 10 monclient: monmap: 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:40.378126+0000 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:40.378126+0000 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.379 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.377+0000 7fa56036d640 1 Processor -- start 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.377+0000 7fa56036d640 1 -- start start 2024-03-25T02:00:40.380 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.377+0000 7fa56036d640 10 monclient: init 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa5580b04a0 0x7fa558146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5580b04a0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 0x7fa558147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient(hunting): picked mon.noname-b con 0x7fa5580b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa558140400 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5580b2410 con 0x7fa558140400 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fa5580b26f0 con 0x7fa5580b04a0 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fa5580b29d0 con 0x7fa5580b2df0 2024-03-25T02:00:40.383 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa56036d640 10 monclient(hunting): authenticate will time out at 1359.568244s 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa55e109640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa5580b04a0 0x7fa558146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa55e109640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa5580b04a0 0x7fa558146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40842/0 (socket says 172.21.15.83:40842) 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa55e109640 1 -- 172.21.15.83:0/2788906517 learned_addr learned my addr 172.21.15.83:0/2788906517 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa55e109640 10 monclient(hunting): get_auth_request con 0x7fa5580b04a0 auth_method 0 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa55e109640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa55e109640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.378+0000 7fa55e109640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.379+0000 7fa55e109640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.379+0000 7fa55e109640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.379+0000 7fa55e109640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55d908640 1 --2- 172.21.15.83:0/2788906517 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 0x7fa558147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.384 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 10 monclient(hunting): handle_auth_done global_id 4411 payload 274 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 1 -- 172.21.15.83:0/2788906517 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 msgr2=0x7fa558147180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 1 --2- 172.21.15.83:0/2788906517 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 0x7fa558147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 1 -- 172.21.15.83:0/2788906517 >> v1:172.21.15.83:6789/0 conn(0x7fa558140400 legacy=0x7fa558140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 10 monclient: _finish_hunting 0 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 1 monclient: found mon.noname-c 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.380+0000 7fa55e109640 1 -- 172.21.15.83:0/2788906517 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5580b3380 con 0x7fa5580b04a0 2024-03-25T02:00:40.385 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.382+0000 7fa55e109640 1 --2- 172.21.15.83:0/2788906517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5580b04a0 0x7fa558146c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa554009870 tx=0x7fa55402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f96398d20e0174a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 1 -- 172.21.15.83:0/2788906517 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554034070 con 0x7fa5580b04a0 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: dump: 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.388 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: _finish_auth 0 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: _check_auth_tickets 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 1 -- 172.21.15.83:0/2788906517 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa554030840 con 0x7fa5580b04a0 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 5 monclient: authenticate success, global_id 4411 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 4 set_mon_vals no callback set 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 1 -- 172.21.15.83:0/2788906517 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa554030c10 con 0x7fa5580b04a0 2024-03-25T02:00:40.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa55d107640 10 monclient: dump: 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.390 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 10 monclient: shutdown 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 1 -- 172.21.15.83:0/2788906517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5580b04a0 msgr2=0x7fa558146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.386+0000 7fa56036d640 1 --2- 172.21.15.83:0/2788906517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5580b04a0 0x7fa558146c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa554009870 tx=0x7fa55402b980 comp rx=0 tx=0).stop 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 1 -- 172.21.15.83:0/2788906517 shutdown_connections 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 1 --2- 172.21.15.83:0/2788906517 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 0x7fa558147180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 1 --2- 172.21.15.83:0/2788906517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5580b04a0 0x7fa558146c40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 1 -- 172.21.15.83:0/2788906517 >> 172.21.15.83:0/2788906517 conn(0x7fa5580b8a00 msgr2=0x7fa5580b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 1 -- 172.21.15.83:0/2788906517 shutdown_connections 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 1 -- 172.21.15.83:0/2788906517 wait complete. 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 10 monclient: build_initial_monmap 2024-03-25T02:00:40.393 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.387+0000 7fa56036d640 10 monclient: monmap: 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 librados: starting msgr at 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 librados: starting objecter 2024-03-25T02:00:40.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 Processor -- start 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 -- start start 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 librados: setting wanted keys 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 librados: calling monclient init 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient: init 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5580b04a0 0x7fa558141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5580b04a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 0x7fa5581dc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5580b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa558140400 0x7fa558133990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): picked mon.noname-b con 0x7fa558140400 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa55814dc70 con 0x7fa5580b04a0 2024-03-25T02:00:40.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa55814daf0 con 0x7fa558140400 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fa55814ddf0 con 0x7fa5580b2df0 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.388+0000 7fa56036d640 10 monclient(hunting): authenticate will time out at 1359.578757s 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa558140400 0x7fa558133990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa558140400 0x7fa558133990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40850/0 (socket says 172.21.15.83:40850) 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 1 -- 172.21.15.83:0/33357039 learned_addr learned my addr 172.21.15.83:0/33357039 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 10 monclient(hunting): get_auth_request con 0x7fa558140400 auth_method 0 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:40.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.389+0000 7fa55e90a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2024-03-25T02:00:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 1 -- 172.21.15.83:0/33357039 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 msgr2=0x7fa5581dc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 0x7fa5581dc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 1 -- 172.21.15.83:0/33357039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5580b04a0 msgr2=0x7fa558141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5580b04a0 0x7fa558141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 10 monclient: _finish_hunting 0 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 1 monclient: found mon.noname-b 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.392+0000 7fa55e90a640 1 -- 172.21.15.83:0/33357039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa558141130 con 0x7fa558140400 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.394+0000 7fa55e90a640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa558140400 0x7fa558133990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c7b0 tx=0x7fa55000cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.394+0000 7fa54effd640 10 client.?.objecter ms_handle_connect 0x7fa558140400 2024-03-25T02:00:40.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.394+0000 7fa54effd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 1 -- 172.21.15.83:0/33357039 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550017070 con 0x7fa558140400 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: dump: 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.401 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: _finish_auth 0 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 1 -- 172.21.15.83:0/33357039 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa550003da0 con 0x7fa558140400 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 1 -- 172.21.15.83:0/33357039 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5500052b0 con 0x7fa558140400 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:40.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa54effd640 10 monclient: dump: 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 5 monclient: authenticate success, global_id 4414 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 10 monclient: _renew_subs 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa55c906640 4 set_mon_vals no callback set 2024-03-25T02:00:40.403 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa558143360 con 0x7fa558140400 2024-03-25T02:00:40.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:40.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 10 monclient: _renew_subs 2024-03-25T02:00:40.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:40.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa55813d070 con 0x7fa558140400 2024-03-25T02:00:40.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.399+0000 7fa56036d640 1 librados: init done 2024-03-25T02:00:40.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.400+0000 7fa56036d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:40.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.400+0000 7fa56036d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:40.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.400+0000 7fa56036d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:40.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.400+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa558075810 con 0x7fa558140400 2024-03-25T02:00:40.621 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.619+0000 7fbdd0ff9640 1 -- 172.21.15.83:0/2502438168 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fbdc40039c0 con 0x7fbdd40b04a0 2024-03-25T02:00:40.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.620+0000 7fbdd0ff9640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbdc006da70 0x7fbdc006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.620+0000 7fbdd37fe640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbdc006da70 0x7fbdc006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.620+0000 7fbdd37fe640 10 monclient: get_auth_request con 0x7fbdc006da70 auth_method 0 2024-03-25T02:00:40.622 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.621+0000 7fbdd0ff9640 10 client.4307.objecter ms_handle_connect 0x7fbdc006da70 2024-03-25T02:00:40.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.621+0000 7fbdd37fe640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbdc006da70 0x7fbdc006ff30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8036000 tx=0x7fbdc803b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:40.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 1 -- 172.21.15.83:0/2502438168 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fbdc40577f0 con 0x7fbdd40b04a0 2024-03-25T02:00:40.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 10 client.4307.objecter ms_dispatch 0x7fbdd4000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:40.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 3 client.4307.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:40.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 3 client.4307.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:40.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-25T02:00:40.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 1 -- 172.21.15.83:0/2502438168 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbdc4059750 con 0x7fbdd40b04a0 2024-03-25T02:00:40.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:40.637 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.634+0000 7fbdd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:40.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.652+0000 7fa54effd640 1 -- 172.21.15.83:0/33357039 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fa550006b10 con 0x7fa558140400 2024-03-25T02:00:40.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.652+0000 7fa54effd640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa53c06da20 0x7fa53c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:40.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.652+0000 7fa55e109640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa53c06da20 0x7fa53c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:40.654 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.652+0000 7fa55e109640 10 monclient: get_auth_request con 0x7fa53c06da20 auth_method 0 2024-03-25T02:00:40.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.653+0000 7fa55e109640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa53c06da20 0x7fa53c06fee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa554036000 tx=0x7fa554030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:40.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.653+0000 7fa54effd640 10 client.4414.objecter ms_handle_connect 0x7fa53c06da20 2024-03-25T02:00:40.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.655+0000 7fa54effd640 1 -- 172.21.15.83:0/33357039 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fa550056a20 con 0x7fa558140400 2024-03-25T02:00:40.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.655+0000 7fa54effd640 10 client.4414.objecter ms_dispatch 0x7fa558000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:40.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.655+0000 7fa54effd640 3 client.4414.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:40.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.655+0000 7fa54effd640 3 client.4414.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:40.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.655+0000 7fa54effd640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T02:00:40.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.667+0000 7fa54effd640 1 -- 172.21.15.83:0/33357039 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa550059080 con 0x7fa558140400 2024-03-25T02:00:40.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.667+0000 7fa54effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:40.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.667+0000 7fa54effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:40.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.831+0000 7fbdda904640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:40.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.831+0000 7fbdda904640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:40.833 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.831+0000 7fbdda904640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:00:40.834 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.831+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fbdd414d8d0 con 0x7fbdd40b04a0 2024-03-25T02:00:40.836 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.835+0000 7fbdd0ff9640 1 -- 172.21.15.83:0/2502438168 <== mon.2 v2:172.21.15.83:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbdc405d7f0 con 0x7fbdd40b04a0 2024-03-25T02:00:40.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.835+0000 7fbdd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-03-25T02:00:40.837 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.835+0000 7fbdd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:40.837 INFO:teuthology.orchestra.run.smithi083.stdout:60129542153 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 10 librados: watch_flush enter 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 10 librados: watch_flush exit 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 10 client.4307.objecter successfully canceled tick 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbdc006da70 msgr2=0x7fbdc006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbdc006da70 0x7fbdc006ff30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8036000 tx=0x7fbdc803b040 comp rx=0 tx=0).stop 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 10 monclient: shutdown 2024-03-25T02:00:40.840 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b04a0 msgr2=0x7fbdd4130c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b04a0 0x7fbdd4130c40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400c7f0 tx=0x7fbdc400ccb0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 shutdown_connections 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbdc006da70 0x7fbdc006ff30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbdd4140400 0x7fbdd4133320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbdd40b2df0 0x7fbdd413de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 --2- 172.21.15.83:0/2502438168 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbdd40b04a0 0x7fbdd4130c40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 >> 172.21.15.83:0/2502438168 conn(0x7fbdd40b8a00 msgr2=0x7fbdd4140850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 shutdown_connections 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 -- 172.21.15.83:0/2502438168 wait complete. 2024-03-25T02:00:40.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.837+0000 7fbdda904640 1 librados: shutdown 2024-03-25T02:00:40.851 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.4 2024-03-25T02:00:40.851 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.856+0000 7fa56036d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.856+0000 7fa56036d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.856+0000 7fa56036d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:40.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.856+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa558141300 con 0x7fa558140400 2024-03-25T02:00:40.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.861+0000 7fa54effd640 1 -- 172.21.15.83:0/33357039 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa550058980 con 0x7fa558140400 2024-03-25T02:00:40.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.861+0000 7fa54effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-03-25T02:00:40.862 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.861+0000 7fa54effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:40.862 INFO:teuthology.orchestra.run.smithi083.stdout:81604378632 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 10 librados: watch_flush enter 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 10 librados: watch_flush exit 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 10 client.4414.objecter successfully canceled tick 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa53c06da20 msgr2=0x7fa53c06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa53c06da20 0x7fa53c06fee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa554036000 tx=0x7fa554030000 comp rx=0 tx=0).stop 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 10 monclient: shutdown 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:40.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa558140400 msgr2=0x7fa558133990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa558140400 0x7fa558133990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c7b0 tx=0x7fa55000cc70 comp rx=0 tx=0).stop 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 shutdown_connections 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fa53c06da20 0x7fa53c06fee0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa558140400 0x7fa558133990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fa5580b2df0 0x7fa5581dc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 --2- 172.21.15.83:0/33357039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa5580b04a0 0x7fa558141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 >> 172.21.15.83:0/33357039 conn(0x7fa5580b8a00 msgr2=0x7fa558146b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 shutdown_connections 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 -- 172.21.15.83:0/33357039 wait complete. 2024-03-25T02:00:40.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:40.863+0000 7fa56036d640 1 librados: shutdown 2024-03-25T02:00:40.876 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378632 got 81604378632 for osd.7 2024-03-25T02:00:40.877 DEBUG:teuthology.parallel:result is None 2024-03-25T02:00:40.877 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-25T02:00:40.877 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-25T02:00:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.014+0000 7f208086a640 10 monclient: get_monmap_and_config 2024-03-25T02:00:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.014+0000 7f208086a640 10 monclient: build_initial_monmap 2024-03-25T02:00:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.014+0000 7f208086a640 10 monclient: monmap: 2024-03-25T02:00:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.016 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:41.015838+0000 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:41.015838+0000 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:41.017 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.014+0000 7f208086a640 1 Processor -- start 2024-03-25T02:00:41.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 1 -- start start 2024-03-25T02:00:41.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient: init 2024-03-25T02:00:41.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:41.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-03-25T02:00:41.019 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f207c06b600 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b9e0 0x7f207c146690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient(hunting): picked mon.noname-b con 0x7f207c06b9e0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f207c073190 0x7f207c072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient(hunting): picked mon.noname-c con 0x7f207c073190 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f207c0b1f90 con 0x7f207c06b600 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f207c140d80 con 0x7f207c073190 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f207c0af020 con 0x7f207c06b9e0 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.015+0000 7f208086a640 10 monclient(hunting): authenticate will time out at 1360.205704s 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f2079d9b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b9e0 0x7f207c146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f207a59c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f207c06b600 0x7f207c0b1880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:45756/0 (socket says 172.21.15.83:45756) 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f207a59c640 1 -- 172.21.15.83:0/4125496367 learned_addr learned my addr 172.21.15.83:0/4125496367 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:00:41.020 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f207ad9d640 1 --2- 172.21.15.83:0/4125496367 >> v2:172.21.15.153:3300/0 conn(0x7f207c073190 0x7f207c072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f2079d9b640 10 monclient(hunting): get_auth_request con 0x7f207c06b9e0 auth_method 0 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f2079d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f2079d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f2079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f207ad9d640 10 monclient(hunting): get_auth_request con 0x7f207c073190 auth_method 0 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f207ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f207ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.016+0000 7f207ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.017+0000 7f2079d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.017+0000 7f2079d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.017+0000 7f2079d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.017+0000 7f207ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.017+0000 7f207ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.021 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.017+0000 7f207ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-03-25T02:00:41.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f207959a640 1 -- 172.21.15.83:0/4125496367 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3797517328 0 0) 0x7f207c0b1f90 con 0x7f207c06b600 2024-03-25T02:00:41.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 1 -- 172.21.15.83:0/4125496367 >> v2:172.21.15.153:3300/0 conn(0x7f207c073190 msgr2=0x7f207c072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:41.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 1 --2- 172.21.15.83:0/4125496367 >> v2:172.21.15.153:3300/0 conn(0x7f207c073190 0x7f207c072870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.022 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 1 -- 172.21.15.83:0/4125496367 >> v1:172.21.15.83:6789/0 conn(0x7f207c06b600 legacy=0x7f207c0b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 10 monclient: _finish_hunting 0 2024-03-25T02:00:41.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 1 monclient: found mon.noname-b 2024-03-25T02:00:41.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:41.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:41.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f2079d9b640 1 -- 172.21.15.83:0/4125496367 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c0af300 con 0x7f207c06b9e0 2024-03-25T02:00:41.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.018+0000 7f207959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-03-25T02:00:41.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.020+0000 7f2079d9b640 1 --2- 172.21.15.83:0/4125496367 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b9e0 0x7f207c146690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2070009d90 tx=0x7f207002bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5acc7354de1f394e server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:41.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 1 -- 172.21.15.83:0/4125496367 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070039070 con 0x7f207c06b9e0 2024-03-25T02:00:41.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:41.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: dump: 2024-03-25T02:00:41.026 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.026 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.026 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: _finish_auth 0 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: _check_auth_tickets 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 1 -- 172.21.15.83:0/4125496367 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f207002cb30 con 0x7f207c06b9e0 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 1 -- 172.21.15.83:0/4125496367 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070034440 con 0x7f207c06b9e0 2024-03-25T02:00:41.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 5 monclient: authenticate success, global_id 4310 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 4 set_mon_vals no callback set 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f207959a640 10 monclient: dump: 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.028 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.029 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 10 monclient: shutdown 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 -- 172.21.15.83:0/4125496367 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b9e0 msgr2=0x7f207c146690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 --2- 172.21.15.83:0/4125496367 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b9e0 0x7f207c146690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2070009d90 tx=0x7f207002bd80 comp rx=0 tx=0).stop 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 -- 172.21.15.83:0/4125496367 shutdown_connections 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 --2- 172.21.15.83:0/4125496367 >> v2:172.21.15.153:3300/0 conn(0x7f207c073190 0x7f207c072870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 --2- 172.21.15.83:0/4125496367 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b9e0 0x7f207c146690 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 -- 172.21.15.83:0/4125496367 >> 172.21.15.83:0/4125496367 conn(0x7f207c0b30a0 msgr2=0x7f207c0b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 -- 172.21.15.83:0/4125496367 shutdown_connections 2024-03-25T02:00:41.032 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.025+0000 7f208086a640 1 -- 172.21.15.83:0/4125496367 wait complete. 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient: build_initial_monmap 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient: monmap: 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 librados: starting msgr at 2024-03-25T02:00:41.033 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 librados: starting objecter 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 Processor -- start 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 -- start start 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 librados: setting wanted keys 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 librados: calling monclient init 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient: init 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b600 0x7f207c13f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): picked mon.noname-c con 0x7f207c06b600 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f207c073190 0x7f207c141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): picked mon.noname-b con 0x7f207c073190 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f207c142ef0 0x7f207c1416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.034 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): picked mon.noname-a con 0x7f207c142ef0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f207c072db0 con 0x7f207c142ef0 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f207c0ae1c0 con 0x7f207c073190 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f207c072f30 con 0x7f207c06b600 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.026+0000 7f208086a640 10 monclient(hunting): authenticate will time out at 1360.216925s 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207a59c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b600 0x7f207c13f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f2079d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f207c073190 0x7f207c141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f207c142ef0 0x7f207c1416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f2079d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f207c073190 0x7f207c141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40864/0 (socket says 172.21.15.83:40864) 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f2079d9b640 1 -- 172.21.15.83:0/2726148147 learned_addr learned my addr 172.21.15.83:0/2726148147 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f2079d9b640 10 monclient(hunting): get_auth_request con 0x7f207c073190 auth_method 0 2024-03-25T02:00:41.035 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f2079d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f2079d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f2079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207a59c640 10 monclient(hunting): get_auth_request con 0x7f207c06b600 auth_method 0 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207ad9d640 10 monclient(hunting): get_auth_request con 0x7f207c142ef0 auth_method 0 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.027+0000 7f207ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f2079d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f2079d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f2079d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f207a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.036 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f207a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f207a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f207ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f207ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.037 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.028+0000 7f207ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 1 -- 172.21.15.83:0/2726148147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b600 msgr2=0x7f207c13f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b600 0x7f207c13f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 1 -- 172.21.15.83:0/2726148147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f207c142ef0 msgr2=0x7f207c1416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f207c142ef0 0x7f207c1416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 10 monclient: _finish_hunting 0 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 1 monclient: found mon.noname-b 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f2079d9b640 1 -- 172.21.15.83:0/2726148147 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c1d86f0 con 0x7f207c073190 2024-03-25T02:00:41.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.031+0000 7f207a59c640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b600 0x7f207c13f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T02:00:41.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.033+0000 7f2079d9b640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f207c073190 0x7f207c141180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2070009d90 tx=0x7f2070004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:41.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.033+0000 7f206affd640 10 client.?.objecter ms_handle_connect 0x7f207c073190 2024-03-25T02:00:41.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.033+0000 7f206affd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:41.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 1 -- 172.21.15.83:0/2726148147 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070044030 con 0x7f207c073190 2024-03-25T02:00:41.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:41.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: dump: 2024-03-25T02:00:41.040 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.040 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: _finish_auth 0 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: _check_auth_tickets 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 5 monclient: authenticate success, global_id 4420 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 1 -- 172.21.15.83:0/2726148147 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f207002c930 con 0x7f207c073190 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:41.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 1 -- 172.21.15.83:0/2726148147 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070039040 con 0x7f207c073190 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 10 monclient: _renew_subs 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207c1d8920 con 0x7f207c073190 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f2078d99640 4 set_mon_vals no callback set 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f206affd640 10 monclient: dump: 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.042 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 10 monclient: _renew_subs 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f207c1d8ec0 con 0x7f207c073190 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.037+0000 7f208086a640 1 librados: init done 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.038+0000 7f208086a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.038+0000 7f208086a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.038+0000 7f208086a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.038+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f207c075810 con 0x7f207c073190 2024-03-25T02:00:41.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.282+0000 7f206affd640 1 -- 172.21.15.83:0/2726148147 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f2070047070 con 0x7f207c073190 2024-03-25T02:00:41.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.282+0000 7f206affd640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f205c06da20 0x7f205c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.282+0000 7f207a59c640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f205c06da20 0x7f205c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.282+0000 7f207a59c640 10 monclient: get_auth_request con 0x7f205c06da20 auth_method 0 2024-03-25T02:00:41.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.283+0000 7f206affd640 10 client.4420.objecter ms_handle_connect 0x7f205c06da20 2024-03-25T02:00:41.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.283+0000 7f207a59c640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f205c06da20 0x7f205c06fee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2074007f70 tx=0x7f2074007f00 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.286+0000 7f206affd640 1 -- 172.21.15.83:0/2726148147 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f207002cae0 con 0x7f207c073190 2024-03-25T02:00:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.286+0000 7f206affd640 10 client.4420.objecter ms_dispatch 0x7f207c000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.286+0000 7f206affd640 3 client.4420.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.286+0000 7f206affd640 3 client.4420.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:41.287 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.286+0000 7f206affd640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T02:00:41.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.296+0000 7f206affd640 1 -- 172.21.15.83:0/2726148147 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f207004e070 con 0x7f207c073190 2024-03-25T02:00:41.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.296+0000 7f206affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:41.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.296+0000 7f206affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:41.524 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.523+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f207c076d20 con 0x7f205c06da20 2024-03-25T02:00:41.526 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.525+0000 7f206affd640 1 -- 172.21.15.83:0/2726148147 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35483 (secure 0 0 0) 0x7f2074001cf0 con 0x7f205c06da20 2024-03-25T02:00:41.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:00:41.527 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-25T02:00:41.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 10 librados: watch_flush enter 2024-03-25T02:00:41.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 10 librados: watch_flush exit 2024-03-25T02:00:41.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T02:00:41.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 10 client.4420.objecter successfully canceled tick 2024-03-25T02:00:41.529 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f205c06da20 msgr2=0x7f205c06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f205c06da20 0x7f205c06fee0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2074007f70 tx=0x7f2074007f00 comp rx=0 tx=0).stop 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 10 monclient: shutdown 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f207c073190 msgr2=0x7f207c141180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f207c073190 0x7f207c141180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2070009d90 tx=0x7f2070004a40 comp rx=0 tx=0).stop 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 shutdown_connections 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f205c06da20 0x7f205c06fee0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.530 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f207c142ef0 0x7f207c1416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f207c073190 0x7f207c141180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 --2- 172.21.15.83:0/2726148147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f207c06b600 0x7f207c13f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.527+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 >> 172.21.15.83:0/2726148147 conn(0x7f207c0b30a0 msgr2=0x7f207c13c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:41.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.528+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 shutdown_connections 2024-03-25T02:00:41.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.528+0000 7f208086a640 1 -- 172.21.15.83:0/2726148147 wait complete. 2024-03-25T02:00:41.531 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.528+0000 7f208086a640 1 librados: shutdown 2024-03-25T02:00:41.539 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-25T02:00:40.286693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218848,"kb_used_data":2672,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755872,"statfs":{"total":773094113280,"available":772870012928,"internally_reserved":0,"allocated":2736128,"data_stored":1464632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.210248"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.637229+0000","last_change":"2024-03-25T02:00:33.801095+0000","last_active":"2024-03-25T02:00:35.637229+0000","last_peered":"2024-03-25T02:00:35.637229+0000","last_clean":"2024-03-25T02:00:35.637229+0000","last_became_active":"2024-03-25T02:00:33.792626+0000","last_became_peered":"2024-03-25T02:00:33.792626+0000","last_unstale":"2024-03-25T02:00:35.637229+0000","last_undegraded":"2024-03-25T02:00:35.637229+0000","last_fullsized":"2024-03-25T02:00:35.637229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:02:09.659440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.319170+0000","last_change":"2024-03-25T02:00:33.937499+0000","last_active":"2024-03-25T02:00:36.319170+0000","last_peered":"2024-03-25T02:00:36.319170+0000","last_clean":"2024-03-25T02:00:36.319170+0000","last_became_active":"2024-03-25T02:00:33.909846+0000","last_became_peered":"2024-03-25T02:00:33.909846+0000","last_unstale":"2024-03-25T02:00:36.319170+0000","last_undegraded":"2024-03-25T02:00:36.319170+0000","last_fullsized":"2024-03-25T02:00:36.319170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:26:26.798933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.634177+0000","last_change":"2024-03-25T02:00:33.514196+0000","last_active":"2024-03-25T02:00:35.634177+0000","last_peered":"2024-03-25T02:00:35.634177+0000","last_clean":"2024-03-25T02:00:35.634177+0000","last_became_active":"2024-03-25T02:00:33.508976+0000","last_became_peered":"2024-03-25T02:00:33.508976+0000","last_unstale":"2024-03-25T02:00:35.634177+0000","last_undegraded":"2024-03-25T02:00:35.634177+0000","last_fullsized":"2024-03-25T02:00:35.634177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:14:11.807682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.307899+0000","last_change":"2024-03-25T02:00:33.702006+0000","last_active":"2024-03-25T02:00:36.307899+0000","last_peered":"2024-03-25T02:00:36.307899+0000","last_clean":"2024-03-25T02:00:36.307899+0000","last_became_active":"2024-03-25T02:00:33.693877+0000","last_became_peered":"2024-03-25T02:00:33.693877+0000","last_unstale":"2024-03-25T02:00:36.307899+0000","last_undegraded":"2024-03-25T02:00:36.307899+0000","last_fullsized":"2024-03-25T02:00:36.307899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:52:31.586659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.626352+0000","last_change":"2024-03-25T02:00:33.707296+0000","last_active":"2024-03-25T02:00:35.626352+0000","last_peered":"2024-03-25T02:00:35.626352+0000","last_clean":"2024-03-25T02:00:35.626352+0000","last_became_active":"2024-03-25T02:00:33.691900+0000","last_became_peered":"2024-03-25T02:00:33.691900+0000","last_unstale":"2024-03-25T02:00:35.626352+0000","last_undegraded":"2024-03-25T02:00:35.626352+0000","last_fullsized":"2024-03-25T02:00:35.626352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:26:57.139959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.314654+0000","last_change":"2024-03-25T02:00:33.459985+0000","last_active":"2024-03-25T02:00:36.314654+0000","last_peered":"2024-03-25T02:00:36.314654+0000","last_clean":"2024-03-25T02:00:36.314654+0000","last_became_active":"2024-03-25T02:00:33.429925+0000","last_became_peered":"2024-03-25T02:00:33.429925+0000","last_unstale":"2024-03-25T02:00:36.314654+0000","last_undegraded":"2024-03-25T02:00:36.314654+0000","last_fullsized":"2024-03-25T02:00:36.314654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:09:49.004802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.618474+0000","last_change":"2024-03-25T02:00:33.383977+0000","last_active":"2024-03-25T02:00:35.618474+0000","last_peered":"2024-03-25T02:00:35.618474+0000","last_clean":"2024-03-25T02:00:35.618474+0000","last_became_active":"2024-03-25T02:00:33.356691+0000","last_became_peered":"2024-03-25T02:00:33.356691+0000","last_unstale":"2024-03-25T02:00:35.618474+0000","last_undegraded":"2024-03-25T02:00:35.618474+0000","last_fullsized":"2024-03-25T02:00:35.618474+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:57.213604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.640166+0000","last_change":"2024-03-25T02:00:33.873073+0000","last_active":"2024-03-25T02:00:35.640166+0000","last_peered":"2024-03-25T02:00:35.640166+0000","last_clean":"2024-03-25T02:00:35.640166+0000","last_became_active":"2024-03-25T02:00:33.865870+0000","last_became_peered":"2024-03-25T02:00:33.865870+0000","last_unstale":"2024-03-25T02:00:35.640166+0000","last_undegraded":"2024-03-25T02:00:35.640166+0000","last_fullsized":"2024-03-25T02:00:35.640166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:52:25.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'77","reported_seq":109,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.252701+0000","last_change":"2024-03-25T02:00:26.482050+0000","last_active":"2024-03-25T02:00:36.252701+0000","last_peered":"2024-03-25T02:00:36.252701+0000","last_clean":"2024-03-25T02:00:36.252701+0000","last_became_active":"2024-03-25T02:00:25.840109+0000","last_became_peered":"2024-03-25T02:00:25.840109+0000","last_unstale":"2024-03-25T02:00:36.252701+0000","last_undegraded":"2024-03-25T02:00:36.252701+0000","last_fullsized":"2024-03-25T02:00:36.252701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_clean_scrub_stamp":"2024-03-25T02:00:15.236797+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:57:04.634251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":20,"seq":85899345931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899345932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":481459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":19,"seq":81604378633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378635,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411335,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":481459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-25T02:00:41.540 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-25T02:00:41.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.641+0000 7fbb7f4d9640 10 monclient: get_monmap_and_config 2024-03-25T02:00:41.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.641+0000 7fbb7f4d9640 10 monclient: build_initial_monmap 2024-03-25T02:00:41.642 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.641+0000 7fbb7f4d9640 10 monclient: monmap: 2024-03-25T02:00:41.642 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:41.642 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.642 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:41.641600+0000 2024-03-25T02:00:41.642 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:41.641600+0000 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:41.643 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.641+0000 7fbb7f4d9640 1 Processor -- start 2024-03-25T02:00:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.641+0000 7fbb7f4d9640 1 -- start start 2024-03-25T02:00:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.641+0000 7fbb7f4d9640 10 monclient: init 2024-03-25T02:00:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-03-25T02:00:41.645 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb780a9b70 addr v1:172.21.15.153:6789/0 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbb780b33f0 0x7fbb78140320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient(hunting): picked mon.noname-c con 0x7fbb780b33f0 addr v2:172.21.15.153:3300/0 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb78140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient(hunting): picked mon.noname-b con 0x7fbb780a9f50 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb780abae0 con 0x7fbb780a9b70 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fbb780abdc0 con 0x7fbb780b33f0 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fbb780ac0a0 con 0x7fbb780a9f50 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.642+0000 7fbb7f4d9640 10 monclient(hunting): authenticate will time out at 1360.831496s 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7da76640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb78140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7da76640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb78140860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:54788/0 (socket says 172.21.15.83:54788) 2024-03-25T02:00:41.646 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7da76640 1 -- 172.21.15.83:0/2832449147 learned_addr learned my addr 172.21.15.83:0/2832449147 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7da76640 10 monclient(hunting): get_auth_request con 0x7fbb780a9f50 auth_method 0 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7da76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7da76640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7da76640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7ca74640 1 --2- 172.21.15.83:0/2832449147 >> v2:172.21.15.153:3300/0 conn(0x7fbb780b33f0 0x7fbb78140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7ca74640 10 monclient(hunting): get_auth_request con 0x7fbb780b33f0 auth_method 0 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7ca74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7ca74640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.643+0000 7fbb7ca74640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.644+0000 7fbb7da76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.644+0000 7fbb7da76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.644+0000 7fbb7da76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.644+0000 7fbb7ca74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.644+0000 7fbb7ca74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.644+0000 7fbb7ca74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2024-03-25T02:00:41.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 1 -- 172.21.15.83:0/2832449147 >> v2:172.21.15.153:3300/0 conn(0x7fbb780b33f0 msgr2=0x7fbb78140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:41.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 1 --2- 172.21.15.83:0/2832449147 >> v2:172.21.15.153:3300/0 conn(0x7fbb780b33f0 0x7fbb78140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 1 -- 172.21.15.83:0/2832449147 >> v1:172.21.15.153:6789/0 conn(0x7fbb780a9b70 legacy=0x7fbb781437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 10 monclient: _finish_hunting 0 2024-03-25T02:00:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 1 monclient: found mon.noname-b 2024-03-25T02:00:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.645+0000 7fbb7da76640 1 -- 172.21.15.83:0/2832449147 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb781432a0 con 0x7fbb780a9f50 2024-03-25T02:00:41.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.647+0000 7fbb7da76640 1 --2- 172.21.15.83:0/2832449147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb78140860 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009870 tx=0x7fbb7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc5c2c6a3d9a5e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:41.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 1 -- 172.21.15.83:0/2832449147 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74034070 con 0x7fbb780a9f50 2024-03-25T02:00:41.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:41.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: dump: 2024-03-25T02:00:41.660 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.660 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.661 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.661 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.661 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.661 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.661 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.661 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.662 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.662 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: _finish_auth 0 2024-03-25T02:00:41.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:41.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:41.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 1 -- 172.21.15.83:0/2832449147 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb74030840 con 0x7fbb780a9f50 2024-03-25T02:00:41.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:41.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 1 -- 172.21.15.83:0/2832449147 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74030c10 con 0x7fbb780a9f50 2024-03-25T02:00:41.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 5 monclient: authenticate success, global_id 4316 2024-03-25T02:00:41.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:41.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 4 set_mon_vals no callback set 2024-03-25T02:00:41.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb6ffff640 10 monclient: dump: 2024-03-25T02:00:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.664 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.665 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.665 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.665 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.665 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.665 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.665 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 10 monclient: shutdown 2024-03-25T02:00:41.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:41.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/2832449147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 msgr2=0x7fbb78140860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/2832449147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb78140860 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009870 tx=0x7fbb7402b980 comp rx=0 tx=0).stop 2024-03-25T02:00:41.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/2832449147 shutdown_connections 2024-03-25T02:00:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/2832449147 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb78140860 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/2832449147 >> v2:172.21.15.153:3300/0 conn(0x7fbb780b33f0 0x7fbb78140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/2832449147 >> 172.21.15.83:0/2832449147 conn(0x7fbb78067e30 msgr2=0x7fbb780b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/2832449147 shutdown_connections 2024-03-25T02:00:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.651+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/2832449147 wait complete. 2024-03-25T02:00:41.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 10 monclient: build_initial_monmap 2024-03-25T02:00:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 10 monclient: monmap: 2024-03-25T02:00:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:41.668 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.669 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:41.669 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:41.669 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:41.669 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 1 librados: starting msgr at 2024-03-25T02:00:41.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 1 librados: starting objecter 2024-03-25T02:00:41.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 1 Processor -- start 2024-03-25T02:00:41.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 1 -- start start 2024-03-25T02:00:41.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 1 librados: setting wanted keys 2024-03-25T02:00:41.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 1 librados: calling monclient init 2024-03-25T02:00:41.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.652+0000 7fbb7f4d9640 10 monclient: init 2024-03-25T02:00:41.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:41.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:00:41.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb780a9b70 0x7fbb780b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): picked mon.noname-b con 0x7fbb780a9b70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:41.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb781dc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): picked mon.noname-c con 0x7fbb780a9f50 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:41.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb780b33f0 0x7fbb781dcb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): picked mon.noname-a con 0x7fbb780b33f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:41.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7814db90 con 0x7fbb780b33f0 2024-03-25T02:00:41.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7814da10 con 0x7fbb780a9b70 2024-03-25T02:00:41.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:41.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fbb7814dd10 con 0x7fbb780a9f50 2024-03-25T02:00:41.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:41.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7f4d9640 10 monclient(hunting): authenticate will time out at 1360.842481s 2024-03-25T02:00:41.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7ca74640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb781dc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.653+0000 7fbb7d275640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb780a9b70 0x7fbb780b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb780b33f0 0x7fbb781dcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb780a9b70 0x7fbb780b7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:40890/0 (socket says 172.21.15.83:40890) 2024-03-25T02:00:41.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 1 -- 172.21.15.83:0/17874563 learned_addr learned my addr 172.21.15.83:0/17874563 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:41.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 10 monclient(hunting): get_auth_request con 0x7fbb780a9b70 auth_method 0 2024-03-25T02:00:41.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7ca74640 10 monclient(hunting): get_auth_request con 0x7fbb780a9f50 auth_method 0 2024-03-25T02:00:41.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7ca74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7ca74640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7ca74640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 10 monclient(hunting): get_auth_request con 0x7fbb780b33f0 auth_method 0 2024-03-25T02:00:41.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:41.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:41.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:41.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.676 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7d275640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7ca74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7ca74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7ca74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:41.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:41.677 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.654+0000 7fbb7da76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:41.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2024-03-25T02:00:41.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 1 -- 172.21.15.83:0/17874563 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 msgr2=0x7fbb781dc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb781dc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 1 -- 172.21.15.83:0/17874563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb780b33f0 msgr2=0x7fbb781dcb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:41.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb780b33f0 0x7fbb781dcb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:41.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 10 monclient: _finish_hunting 0 2024-03-25T02:00:41.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 1 monclient: found mon.noname-b 2024-03-25T02:00:41.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:41.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.657+0000 7fbb7d275640 1 -- 172.21.15.83:0/17874563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb780b9150 con 0x7fbb780a9b70 2024-03-25T02:00:41.681 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.659+0000 7fbb7d275640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb780a9b70 0x7fbb780b7690 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000c790 tx=0x7fbb7000cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.659+0000 7fbb6dffb640 10 client.?.objecter ms_handle_connect 0x7fbb780a9b70 2024-03-25T02:00:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.659+0000 7fbb6dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 1 -- 172.21.15.83:0/17874563 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70016020 con 0x7fbb780a9b70 2024-03-25T02:00:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.682 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:41.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: dump: 2024-03-25T02:00:41.683 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.683 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.683 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.683 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.683 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.684 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.684 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.684 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.684 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.684 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:41.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:41.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:41.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 1 -- 172.21.15.83:0/17874563 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb70006d50 con 0x7fbb780a9b70 2024-03-25T02:00:41.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:41.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 1 -- 172.21.15.83:0/17874563 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70003bb0 con 0x7fbb780a9b70 2024-03-25T02:00:41.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 5 monclient: authenticate success, global_id 4429 2024-03-25T02:00:41.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 10 monclient: _renew_subs 2024-03-25T02:00:41.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb780b97a0 con 0x7fbb780a9b70 2024-03-25T02:00:41.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:41.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6f7fe640 4 set_mon_vals no callback set 2024-03-25T02:00:41.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:00:41.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb6dffb640 10 monclient: dump: 2024-03-25T02:00:41.687 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:41.687 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:41.688 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.688 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:41.688 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:41.688 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:41.688 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:41.688 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:41.689 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:41.689 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:41.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:41.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 10 monclient: _renew_subs 2024-03-25T02:00:41.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb781dd060 con 0x7fbb780a9b70 2024-03-25T02:00:41.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.663+0000 7fbb7f4d9640 1 librados: init done 2024-03-25T02:00:41.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.664+0000 7fbb7f4d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:41.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.664+0000 7fbb7f4d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:41.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.664+0000 7fbb7f4d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:00:41.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.664+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb78075810 con 0x7fbb780a9b70 2024-03-25T02:00:41.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.682+0000 7fbb6dffb640 1 -- 172.21.15.83:0/17874563 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fbb7000a850 con 0x7fbb780a9b70 2024-03-25T02:00:41.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.682+0000 7fbb6dffb640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbb5c06da70 0x7fbb5c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:41.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.683+0000 7fbb7ca74640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbb5c06da70 0x7fbb5c06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:41.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.683+0000 7fbb7ca74640 10 monclient: get_auth_request con 0x7fbb5c06da70 auth_method 0 2024-03-25T02:00:41.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.683+0000 7fbb7ca74640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbb5c06da70 0x7fbb5c06ff30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb7813de80 tx=0x7fbb68009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:41.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.683+0000 7fbb6dffb640 10 client.4429.objecter ms_handle_connect 0x7fbb5c06da70 2024-03-25T02:00:41.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.684+0000 7fbb6dffb640 1 -- 172.21.15.83:0/17874563 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fbb70057d90 con 0x7fbb780a9b70 2024-03-25T02:00:41.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.684+0000 7fbb6dffb640 10 client.4429.objecter ms_dispatch 0x7fbb78000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:41.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.684+0000 7fbb6dffb640 3 client.4429.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:41.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.684+0000 7fbb6dffb640 3 client.4429.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:41.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.685+0000 7fbb6dffb640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T02:00:41.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.925+0000 7fbb6dffb640 1 -- 172.21.15.83:0/17874563 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbb70020020 con 0x7fbb780a9b70 2024-03-25T02:00:41.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.925+0000 7fbb6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:41.927 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:41.925+0000 7fbb6dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:42.096 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.095+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb78076d20 con 0x7fbb5c06da70 2024-03-25T02:00:42.098 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.096+0000 7fbb6dffb640 1 -- 172.21.15.83:0/17874563 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35483 (secure 0 0 0) 0x7fbb680119d0 con 0x7fbb5c06da70 2024-03-25T02:00:42.099 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:00:42.099 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-25T02:00:42.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 10 librados: watch_flush enter 2024-03-25T02:00:42.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 10 librados: watch_flush exit 2024-03-25T02:00:42.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T02:00:42.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 10 client.4429.objecter successfully canceled tick 2024-03-25T02:00:42.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbb5c06da70 msgr2=0x7fbb5c06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbb5c06da70 0x7fbb5c06ff30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb7813de80 tx=0x7fbb68009190 comp rx=0 tx=0).stop 2024-03-25T02:00:42.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 10 monclient: shutdown 2024-03-25T02:00:42.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:42.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb780a9b70 msgr2=0x7fbb780b7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb780a9b70 0x7fbb780b7690 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000c790 tx=0x7fbb7000cc50 comp rx=0 tx=0).stop 2024-03-25T02:00:42.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 shutdown_connections 2024-03-25T02:00:42.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fbb5c06da70 0x7fbb5c06ff30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb780b33f0 0x7fbb781dcb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fbb780a9f50 0x7fbb781dc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 --2- 172.21.15.83:0/17874563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb780a9b70 0x7fbb780b7690 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 >> 172.21.15.83:0/17874563 conn(0x7fbb78067e30 msgr2=0x7fbb7813e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 shutdown_connections 2024-03-25T02:00:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 -- 172.21.15.83:0/17874563 wait complete. 2024-03-25T02:00:42.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.099+0000 7fbb7f4d9640 1 librados: shutdown 2024-03-25T02:00:42.111 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-25T02:00:40.286693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218848,"kb_used_data":2672,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755872,"statfs":{"total":773094113280,"available":772870012928,"internally_reserved":0,"allocated":2736128,"data_stored":1464632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.210248"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.637229+0000","last_change":"2024-03-25T02:00:33.801095+0000","last_active":"2024-03-25T02:00:35.637229+0000","last_peered":"2024-03-25T02:00:35.637229+0000","last_clean":"2024-03-25T02:00:35.637229+0000","last_became_active":"2024-03-25T02:00:33.792626+0000","last_became_peered":"2024-03-25T02:00:33.792626+0000","last_unstale":"2024-03-25T02:00:35.637229+0000","last_undegraded":"2024-03-25T02:00:35.637229+0000","last_fullsized":"2024-03-25T02:00:35.637229+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:02:09.659440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.319170+0000","last_change":"2024-03-25T02:00:33.937499+0000","last_active":"2024-03-25T02:00:36.319170+0000","last_peered":"2024-03-25T02:00:36.319170+0000","last_clean":"2024-03-25T02:00:36.319170+0000","last_became_active":"2024-03-25T02:00:33.909846+0000","last_became_peered":"2024-03-25T02:00:33.909846+0000","last_unstale":"2024-03-25T02:00:36.319170+0000","last_undegraded":"2024-03-25T02:00:36.319170+0000","last_fullsized":"2024-03-25T02:00:36.319170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:26:26.798933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.634177+0000","last_change":"2024-03-25T02:00:33.514196+0000","last_active":"2024-03-25T02:00:35.634177+0000","last_peered":"2024-03-25T02:00:35.634177+0000","last_clean":"2024-03-25T02:00:35.634177+0000","last_became_active":"2024-03-25T02:00:33.508976+0000","last_became_peered":"2024-03-25T02:00:33.508976+0000","last_unstale":"2024-03-25T02:00:35.634177+0000","last_undegraded":"2024-03-25T02:00:35.634177+0000","last_fullsized":"2024-03-25T02:00:35.634177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:14:11.807682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.307899+0000","last_change":"2024-03-25T02:00:33.702006+0000","last_active":"2024-03-25T02:00:36.307899+0000","last_peered":"2024-03-25T02:00:36.307899+0000","last_clean":"2024-03-25T02:00:36.307899+0000","last_became_active":"2024-03-25T02:00:33.693877+0000","last_became_peered":"2024-03-25T02:00:33.693877+0000","last_unstale":"2024-03-25T02:00:36.307899+0000","last_undegraded":"2024-03-25T02:00:36.307899+0000","last_fullsized":"2024-03-25T02:00:36.307899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:52:31.586659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.626352+0000","last_change":"2024-03-25T02:00:33.707296+0000","last_active":"2024-03-25T02:00:35.626352+0000","last_peered":"2024-03-25T02:00:35.626352+0000","last_clean":"2024-03-25T02:00:35.626352+0000","last_became_active":"2024-03-25T02:00:33.691900+0000","last_became_peered":"2024-03-25T02:00:33.691900+0000","last_unstale":"2024-03-25T02:00:35.626352+0000","last_undegraded":"2024-03-25T02:00:35.626352+0000","last_fullsized":"2024-03-25T02:00:35.626352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:26:57.139959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.314654+0000","last_change":"2024-03-25T02:00:33.459985+0000","last_active":"2024-03-25T02:00:36.314654+0000","last_peered":"2024-03-25T02:00:36.314654+0000","last_clean":"2024-03-25T02:00:36.314654+0000","last_became_active":"2024-03-25T02:00:33.429925+0000","last_became_peered":"2024-03-25T02:00:33.429925+0000","last_unstale":"2024-03-25T02:00:36.314654+0000","last_undegraded":"2024-03-25T02:00:36.314654+0000","last_fullsized":"2024-03-25T02:00:36.314654+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:09:49.004802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.618474+0000","last_change":"2024-03-25T02:00:33.383977+0000","last_active":"2024-03-25T02:00:35.618474+0000","last_peered":"2024-03-25T02:00:35.618474+0000","last_clean":"2024-03-25T02:00:35.618474+0000","last_became_active":"2024-03-25T02:00:33.356691+0000","last_became_peered":"2024-03-25T02:00:33.356691+0000","last_unstale":"2024-03-25T02:00:35.618474+0000","last_undegraded":"2024-03-25T02:00:35.618474+0000","last_fullsized":"2024-03-25T02:00:35.618474+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:57.213604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:35.640166+0000","last_change":"2024-03-25T02:00:33.873073+0000","last_active":"2024-03-25T02:00:35.640166+0000","last_peered":"2024-03-25T02:00:35.640166+0000","last_clean":"2024-03-25T02:00:35.640166+0000","last_became_active":"2024-03-25T02:00:33.865870+0000","last_became_peered":"2024-03-25T02:00:33.865870+0000","last_unstale":"2024-03-25T02:00:35.640166+0000","last_undegraded":"2024-03-25T02:00:35.640166+0000","last_fullsized":"2024-03-25T02:00:35.640166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:52:25.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"23'77","reported_seq":109,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-25T02:00:36.252701+0000","last_change":"2024-03-25T02:00:26.482050+0000","last_active":"2024-03-25T02:00:36.252701+0000","last_peered":"2024-03-25T02:00:36.252701+0000","last_clean":"2024-03-25T02:00:36.252701+0000","last_became_active":"2024-03-25T02:00:25.840109+0000","last_became_peered":"2024-03-25T02:00:25.840109+0000","last_unstale":"2024-03-25T02:00:36.252701+0000","last_undegraded":"2024-03-25T02:00:36.252701+0000","last_fullsized":"2024-03-25T02:00:36.252701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_clean_scrub_stamp":"2024-03-25T02:00:15.236797+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:57:04.634251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":20,"seq":85899345931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899345932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":481459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":19,"seq":81604378633,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378635,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411335,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":481459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":83619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-25T02:00:42.112 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-25T02:00:42.112 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-25T02:00:42.113 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-25T02:00:42.113 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-25T02:00:42.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.212+0000 7f24dd8f6640 10 monclient: get_monmap_and_config 2024-03-25T02:00:42.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.212+0000 7f24dd8f6640 10 monclient: build_initial_monmap 2024-03-25T02:00:42.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.212+0000 7f24dd8f6640 10 monclient: monmap: 2024-03-25T02:00:42.213 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:42.213 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:42.213 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:00:42.212911+0000 2024-03-25T02:00:42.213 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:00:42.212911+0000 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:00:42.214 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:42.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 1 Processor -- start 2024-03-25T02:00:42.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 1 -- start start 2024-03-25T02:00:42.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient: init 2024-03-25T02:00:42.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:42.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-03-25T02:00:42.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d80a9c40 0x7f24d80aa020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:42.216 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient(hunting): picked mon.noname-b con 0x7f24d80a9c40 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f24d80aaec0 0x7f24d8143730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient(hunting): picked mon.noname-a con 0x7f24d80aaec0 addr v2:172.21.15.83:3300/0 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24d8143c70 addr v1:172.21.15.83:6789/0 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24d80abf00 con 0x7f24d8143c70 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f24d81432a0 con 0x7f24d80aaec0 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f24d80ac170 con 0x7f24d80a9c40 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.213+0000 7f24dd8f6640 10 monclient(hunting): authenticate will time out at 1361.402610s 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d67fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f24d80aaec0 0x7f24d8143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d6ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d80a9c40 0x7f24d80aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d67fc640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f24d80aaec0 0x7f24d8143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56272/0 (socket says 172.21.15.83:56272) 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d67fc640 1 -- 172.21.15.83:0/3033080742 learned_addr learned my addr 172.21.15.83:0/3033080742 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d67fc640 10 monclient(hunting): get_auth_request con 0x7f24d80aaec0 auth_method 0 2024-03-25T02:00:42.217 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d6ffd640 10 monclient(hunting): get_auth_request con 0x7f24d80a9c40 auth_method 0 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.214+0000 7f24d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.215+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.215+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.215+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.215+0000 7f24d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.215+0000 7f24d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:42.218 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.215+0000 7f24d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 1 -- 172.21.15.83:0/3033080742 >> v2:172.21.15.83:3300/0 conn(0x7f24d80aaec0 msgr2=0x7f24d8143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 1 --2- 172.21.15.83:0/3033080742 >> v2:172.21.15.83:3300/0 conn(0x7f24d80aaec0 0x7f24d8143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 1 -- 172.21.15.83:0/3033080742 >> v1:172.21.15.83:6789/0 conn(0x7f24d8143c70 legacy=0x7f24d81403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 1 monclient: found mon.noname-b 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:00:42.219 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.216+0000 7f24d6ffd640 1 -- 172.21.15.83:0/3033080742 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d80b3490 con 0x7f24d80a9c40 2024-03-25T02:00:42.220 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.218+0000 7f24d6ffd640 1 --2- 172.21.15.83:0/3033080742 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d80a9c40 0x7f24d80aa020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24c80097a0 tx=0x7f24c802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba8d9821e1ca2c60 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:42.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24d5ffb640 1 -- 172.21.15.83:0/3033080742 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c80315a0 con 0x7f24d80a9c40 2024-03-25T02:00:42.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:42.221 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24d5ffb640 10 monclient: dump: 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24d5ffb640 10 monclient: _finish_auth 0 2024-03-25T02:00:42.222 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:00:42.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:42.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24dd8f6640 5 monclient: authenticate success, global_id 4322 2024-03-25T02:00:42.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.220+0000 7f24dd8f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:00:42.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24d5ffb640 1 -- 172.21.15.83:0/3033080742 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c8037070 con 0x7f24d80a9c40 2024-03-25T02:00:42.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24d5ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:42.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24d5ffb640 1 -- 172.21.15.83:0/3033080742 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8031960 con 0x7f24d80a9c40 2024-03-25T02:00:42.223 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24dd8f6640 10 monclient: get_monmap_and_config success 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24dd8f6640 4 set_mon_vals no callback set 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.222+0000 7f24d5ffb640 10 monclient: dump: 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:42.224 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 10 monclient: shutdown 2024-03-25T02:00:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 -- 172.21.15.83:0/3033080742 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d80a9c40 msgr2=0x7f24d80aa020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/3033080742 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d80a9c40 0x7f24d80aa020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24c80097a0 tx=0x7f24c802b840 comp rx=0 tx=0).stop 2024-03-25T02:00:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 -- 172.21.15.83:0/3033080742 shutdown_connections 2024-03-25T02:00:42.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/3033080742 >> v2:172.21.15.83:3300/0 conn(0x7f24d80aaec0 0x7f24d8143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/3033080742 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d80a9c40 0x7f24d80aa020 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 -- 172.21.15.83:0/3033080742 >> 172.21.15.83:0/3033080742 conn(0x7f24d80b8a90 msgr2=0x7f24d80b8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 -- 172.21.15.83:0/3033080742 shutdown_connections 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.223+0000 7f24dd8f6640 1 -- 172.21.15.83:0/3033080742 wait complete. 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 10 monclient: build_initial_monmap 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 10 monclient: monmap: 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:00:42.230 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 1 librados: starting msgr at 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 1 librados: starting objecter 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 1 Processor -- start 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 1 -- start start 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 1 librados: setting wanted keys 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 1 librados: calling monclient init 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.224+0000 7f24dd8f6640 10 monclient: init 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient: _reopen_session rank -1 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24d80a9c40 0x7f24d8146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): picked mon.noname-a con 0x7f24d80a9c40 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:00:42.231 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24d80aaec0 0x7f24d8144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): picked mon.noname-b con 0x7f24d80aaec0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d8143c70 0x7f24d81391e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): picked mon.noname-c con 0x7f24d8143c70 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f24d814d9c0 con 0x7f24d80a9c40 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f24d8140d80 con 0x7f24d80aaec0 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): start opening mon connection 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f24d814db40 con 0x7f24d8143c70 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): _renew_subs 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.225+0000 7f24dd8f6640 10 monclient(hunting): authenticate will time out at 1361.414390s 2024-03-25T02:00:42.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24d80a9c40 0x7f24d8146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24d80a9c40 0x7f24d8146ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:56282/0 (socket says 172.21.15.83:56282) 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 1 -- 172.21.15.83:0/2127730892 learned_addr learned my addr 172.21.15.83:0/2127730892 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 10 monclient(hunting): get_auth_request con 0x7f24d80a9c40 auth_method 0 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.226+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.227+0000 7f24d67fc640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24d80aaec0 0x7f24d8144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:42.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.227+0000 7f24d77fe640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d8143c70 0x7f24d81391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.227+0000 7f24d67fc640 10 monclient(hunting): get_auth_request con 0x7f24d80aaec0 auth_method 0 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.227+0000 7f24d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.227+0000 7f24d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.227+0000 7f24d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d77fe640 10 monclient(hunting): get_auth_request con 0x7f24d8143c70 auth_method 0 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:00:42.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:00:42.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.228+0000 7f24d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:00:42.235 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 1 -- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d8143c70 msgr2=0x7f24d81391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d8143c70 0x7f24d81391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 1 -- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24d80aaec0 msgr2=0x7f24d8144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24d80aaec0 0x7f24d8144690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 1 monclient: found mon.noname-a 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.229+0000 7f24d6ffd640 1 -- 172.21.15.83:0/2127730892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d813a670 con 0x7f24d80a9c40 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.232+0000 7f24d6ffd640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24d80a9c40 0x7f24d8146ff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24c8034040 tx=0x7f24c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.232+0000 7f24bffff640 10 client.?.objecter ms_handle_connect 0x7f24d80a9c40 2024-03-25T02:00:42.236 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.232+0000 7f24bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:00:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 1 -- 172.21.15.83:0/2127730892 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8032630 con 0x7f24d80a9c40 2024-03-25T02:00:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:00:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: dump: 2024-03-25T02:00:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:42.239 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: _finish_auth 0 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: _check_auth_tickets 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 1 -- 172.21.15.83:0/2127730892 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c80327f0 con 0x7f24d80a9c40 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 1 -- 172.21.15.83:0/2127730892 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8032be0 con 0x7f24d80a9c40 2024-03-25T02:00:42.240 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 5 monclient: authenticate success, global_id 4389 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 10 monclient: _renew_subs 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d813a950 con 0x7f24d80a9c40 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24d57fa640 4 set_mon_vals no callback set 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24bffff640 10 monclient: dump: 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:00:42.241 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:00:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:00:42.242 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:00:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:00:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 10 monclient: _renew_subs 2024-03-25T02:00:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d8144bd0 con 0x7f24d80a9c40 2024-03-25T02:00:42.242 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.238+0000 7f24dd8f6640 1 librados: init done 2024-03-25T02:00:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.239+0000 7f24dd8f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.239+0000 7f24dd8f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.239+0000 7f24dd8f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:42.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.239+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d8075810 con 0x7f24d80a9c40 2024-03-25T02:00:42.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.257+0000 7f24bffff640 1 -- 172.21.15.83:0/2127730892 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f24c804e3f0 con 0x7f24d80a9c40 2024-03-25T02:00:42.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.257+0000 7f24bffff640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f24c406da70 0x7f24c406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:00:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.258+0000 7f24d67fc640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f24c406da70 0x7f24c406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:00:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.258+0000 7f24d67fc640 10 monclient: get_auth_request con 0x7f24c406da70 auth_method 0 2024-03-25T02:00:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.258+0000 7f24d67fc640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f24c406da70 0x7f24c406ff30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24d813d500 tx=0x7f24cc009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:00:42.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.258+0000 7f24bffff640 10 client.4389.objecter ms_handle_connect 0x7f24c406da70 2024-03-25T02:00:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.261+0000 7f24bffff640 1 -- 172.21.15.83:0/2127730892 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f24c802f070 con 0x7f24d80a9c40 2024-03-25T02:00:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.261+0000 7f24bffff640 10 client.4389.objecter ms_dispatch 0x7f24d8000b90 osd_map(25..25 src has 1..25) 2024-03-25T02:00:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.261+0000 7f24bffff640 3 client.4389.objecter handle_osd_map got epochs [25,25] > 0 2024-03-25T02:00:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.261+0000 7f24bffff640 3 client.4389.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:00:42.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.261+0000 7f24bffff640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T02:00:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.498+0000 7f24bffff640 1 -- 172.21.15.83:0/2127730892 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f24c8040070 con 0x7f24d80a9c40 2024-03-25T02:00:42.499 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.498+0000 7f24bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:00:42.500 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.498+0000 7f24bffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:00:42.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.737+0000 7f24dd8f6640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-03-25T02:00:42.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.737+0000 7f24dd8f6640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-03-25T02:00:42.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.737+0000 7f24dd8f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:00:42.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.737+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f24d8073b30 con 0x7f24d80a9c40 2024-03-25T02:00:42.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.760+0000 7f24bffff640 1 -- 172.21.15.83:0/2127730892 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f24c807b150 con 0x7f24d80a9c40 2024-03-25T02:00:42.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.760+0000 7f24bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-03-25T02:00:42.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.760+0000 7f24bffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:00:42.761 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:00:42.761 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-25T02:00:42.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 10 librados: watch_flush enter 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 10 librados: watch_flush exit 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 10 client.4389.objecter successfully canceled tick 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f24c406da70 msgr2=0x7f24c406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f24c406da70 0x7f24c406ff30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f24d813d500 tx=0x7f24cc009240 comp rx=0 tx=0).stop 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 10 monclient: shutdown 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24d80a9c40 msgr2=0x7f24d8146ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24d80a9c40 0x7f24d8146ff0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24c8034040 tx=0x7f24c80046c0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 shutdown_connections 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f24c406da70 0x7f24c406ff30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24d8143c70 0x7f24d81391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24d80aaec0 0x7f24d8144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 --2- 172.21.15.83:0/2127730892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24d80a9c40 0x7f24d8146ff0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:00:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 >> 172.21.15.83:0/2127730892 conn(0x7f24d80b8a90 msgr2=0x7f24d813f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:00:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 shutdown_connections 2024-03-25T02:00:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 -- 172.21.15.83:0/2127730892 wait complete. 2024-03-25T02:00:42.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:00:42.762+0000 7f24dd8f6640 1 librados: shutdown 2024-03-25T02:00:42.775 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-25T02:00:42.775 INFO:teuthology.run_tasks:Running task workunit... 2024-03-25T02:00:42.787 INFO:tasks.workunit:Pulling workunits from ref 9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-25T02:00:42.788 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-25T02:00:42.788 DEBUG:teuthology.orchestra.run.smithi083:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-25T02:00:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T02:00:42.802 INFO:teuthology.orchestra.run.smithi083.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-25T02:00:42.802 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-25T02:00:42.857 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-25T02:00:42.857 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-25T02:00:42.912 INFO:tasks.workunit:timeout=6h 2024-03-25T02:00:42.912 INFO:tasks.workunit:cleanup=True 2024-03-25T02:00:42.913 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-25T02:00:42.967 INFO:tasks.workunit.client.0.smithi083.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-25T02:00:46.264 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:29.100 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.270 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:29.106 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.324 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:29.117 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.330 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:29.122 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.343 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:29.179 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.348 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:29.184 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.350 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:29.186 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.395 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:29.187 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:00:46.399 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:29.235 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.273 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:49.108 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.356 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:49.192 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.382 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:01:49.218 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.406 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:49.198 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.414 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:49.206 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.430 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:01:49.222 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.452 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:49.287 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.457 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:49.293 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:06.480 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:01:49.316 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.348 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:09.183 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.404 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:09.240 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.421 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:09.257 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.461 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:09.297 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.463 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:09.298 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.463 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:09.256 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.470 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:09.262 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.481 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:09.317 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:26.485 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:09.277 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr:Note: switching to '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'. 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr:state without impacting any branches by switching back to a branch. 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-25T02:01:32.671 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr: git switch -c 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr:Or undo this operation with: 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr: git switch - 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr: 2024-03-25T02:01:32.672 INFO:tasks.workunit.client.0.smithi083.stderr:HEAD is now at 9c429fb6abf Merge branch 'wip-osd-ec-partial-reads' of https://github.com/rzarzynski/ceph into wip-yuri10-testing-2024-03-24-1159 2024-03-25T02:01:32.676 DEBUG:teuthology.orchestra.run.smithi083:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-25T02:01:32.743 INFO:tasks.workunit.client.0.smithi083.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-25T02:01:32.745 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-25T02:01:32.745 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-25T02:01:33.414 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-25T02:01:33.504 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-25T02:01:33.543 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-25T02:01:33.544 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-25T02:01:33.544 INFO:tasks.workunit.client.0.smithi083.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-25T02:01:33.582 INFO:tasks.workunit.client.0.smithi083.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-25T02:01:33.585 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-25T02:01:33.586 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-25T02:01:33.641 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-25T02:01:33.643 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-25T02:01:33.644 DEBUG:teuthology.orchestra.run.smithi083:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9c429fb6abfb5deed62697d56d8997d0f0d6d83f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ parallel=1 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' '' = --serial ']' 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ crimson=0 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' '' = --crimson ']' 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ color= 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -t 1 ']' 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-25T02:01:33.702 INFO:tasks.workunit.client.0.smithi083.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:+ declare -A pids 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_aio 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_aio' 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_aio 2024-03-25T02:01:33.703 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.704 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_aio 2024-03-25T02:01:33.705 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40622 2024-03-25T02:01:33.705 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_aio on pid 40622' 2024-03-25T02:01:33.705 INFO:tasks.workunit.client.0.smithi083.stdout:test api_aio on pid 40622 2024-03-25T02:01:33.709 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40622 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_aio_pp 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_aio_pp' 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_aio_pp 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.710 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.711 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.712 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.713 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.713 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.713 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.713 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.713 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.713 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.713 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_aio_pp 2024-03-25T02:01:33.714 INFO:tasks.workunit.client.0.smithi083.stdout:test api_aio_pp on pid 40629 2024-03-25T02:01:33.718 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40629 2024-03-25T02:01:33.718 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_aio_pp on pid 40629' 2024-03-25T02:01:33.718 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40629 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_io 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_io' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_io 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.719 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.720 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.721 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_io 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40638 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_io on pid 40638' 2024-03-25T02:01:33.722 INFO:tasks.workunit.client.0.smithi083.stdout:test api_io on pid 40638 2024-03-25T02:01:33.727 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40638 2024-03-25T02:01:33.727 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.727 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.727 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-25T02:01:33.727 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_io_pp 2024-03-25T02:01:33.727 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_io_pp' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_io_pp 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.728 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.729 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.730 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.731 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.732 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.732 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.732 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_io_pp 2024-03-25T02:01:33.732 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40649 2024-03-25T02:01:33.732 INFO:tasks.workunit.client.0.smithi083.stdout:test api_io_pp on pid 40649 2024-03-25T02:01:33.738 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_io_pp on pid 40649' 2024-03-25T02:01:33.738 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40649 2024-03-25T02:01:33.738 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.738 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-25T02:01:33.738 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_asio 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_asio' 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_asio 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.739 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.740 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.741 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.742 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.743 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_asio 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40663 2024-03-25T02:01:33.744 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_asio on pid 40663' 2024-03-25T02:01:33.745 INFO:tasks.workunit.client.0.smithi083.stdout:test api_asio on pid 40663 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40663 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_list 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_list' 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.752 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_list 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.753 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.754 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.755 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.756 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.757 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40622/exe 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.758 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_list 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40680 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.759 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-25T02:01:33.760 INFO:tasks.workunit.client.0.smithi083.stdout:test api_list on pid 40680 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_list on pid 40680' 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40680 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_lock 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_lock' 2024-03-25T02:01:33.769 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.770 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.771 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.772 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_lock 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_aio: /' 2024-03-25T02:01:33.773 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.774 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.775 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.776 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.777 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40629/exe 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.778 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.779 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_lock 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40704 2024-03-25T02:01:33.780 INFO:tasks.workunit.client.0.smithi083.stdout:test api_lock on pid 40704 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_lock on pid 40704' 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40704 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.789 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.790 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.791 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40638/exe 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_lock_pp 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_lock_pp' 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.792 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.793 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_lock_pp 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.794 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.795 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_io.log 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_lock_pp 2024-03-25T02:01:33.796 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40724 2024-03-25T02:01:33.797 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_lock_pp on pid 40724' 2024-03-25T02:01:33.797 INFO:tasks.workunit.client.0.smithi083.stdout:test api_lock_pp on pid 40724 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40724 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.808 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.809 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.810 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_io: /' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.811 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.812 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40649/exe 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.813 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_misc 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_misc' 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.814 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.815 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_misc 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.816 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.817 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_misc 2024-03-25T02:01:33.818 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40748 2024-03-25T02:01:33.819 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_misc on pid 40748' 2024-03-25T02:01:33.819 INFO:tasks.workunit.client.0.smithi083.stdout:test api_misc on pid 40748 2024-03-25T02:01:33.826 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40748 2024-03-25T02:01:33.826 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.826 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_misc_pp 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.827 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_misc_pp' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_misc_pp 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.828 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.829 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.830 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.831 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40663/exe 2024-03-25T02:01:33.832 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.833 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.833 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.833 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_misc_pp 2024-03-25T02:01:33.833 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40765 2024-03-25T02:01:33.833 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_misc_pp on pid 40765' 2024-03-25T02:01:33.833 INFO:tasks.workunit.client.0.smithi083.stdout:test api_misc_pp on pid 40765 2024-03-25T02:01:33.842 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40765 2024-03-25T02:01:33.842 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.842 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.842 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-25T02:01:33.842 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.842 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_tier_pp 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_tier_pp' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.843 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.844 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_asio: /' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.845 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.846 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.847 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.848 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_tier_pp 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_tier_pp 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40781 2024-03-25T02:01:33.849 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40704/exe 2024-03-25T02:01:33.850 INFO:tasks.workunit.client.0.smithi083.stdout:test api_tier_pp on pid 40781 2024-03-25T02:01:33.866 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_tier_pp on pid 40781' 2024-03-25T02:01:33.866 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40781 2024-03-25T02:01:33.866 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.867 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.867 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.867 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-25T02:01:33.867 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.867 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.867 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.867 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_pool 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_pool' 2024-03-25T02:01:33.868 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.869 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.870 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.871 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.872 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.873 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_lock: /' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.874 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_pool 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.875 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.876 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_pool 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40806 2024-03-25T02:01:33.877 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_pool on pid 40806' 2024-03-25T02:01:33.878 INFO:tasks.workunit.client.0.smithi083.stdout:test api_pool on pid 40806 2024-03-25T02:01:33.894 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40806 2024-03-25T02:01:33.894 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.894 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.894 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-25T02:01:33.894 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.895 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.895 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.895 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.895 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.895 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.895 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.896 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.897 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.898 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.899 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.900 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.900 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.900 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.900 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.900 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.900 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.901 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.901 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.901 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.901 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.901 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.901 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.902 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.902 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.902 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.902 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.902 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.902 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.903 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.903 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.903 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.903 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.903 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.904 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40680/exe 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40748/exe 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_snapshots 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.905 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_snapshots' 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.906 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.907 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.908 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.909 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_misc: /' 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_snapshots 2024-03-25T02:01:33.910 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.911 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40724/exe 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_list.log 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.912 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_list: /' 2024-03-25T02:01:33.913 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-25T02:01:33.914 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_snapshots 2024-03-25T02:01:33.914 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40863 2024-03-25T02:01:33.914 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_snapshots on pid 40863' 2024-03-25T02:01:33.914 INFO:tasks.workunit.client.0.smithi083.stdout:test api_snapshots on pid 40863 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40863 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_snapshots_pp 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.918 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.919 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_snapshots_pp' 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_snapshots_pp 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_snapshots_pp 2024-03-25T02:01:33.920 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40882 2024-03-25T02:01:33.921 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_snapshots_pp on pid 40882' 2024-03-25T02:01:33.921 INFO:tasks.workunit.client.0.smithi083.stdout:test api_snapshots_pp on pid 40882 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40882 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_stat 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_stat' 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.937 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.938 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.939 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.941 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.941 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.941 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.941 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.941 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.941 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.942 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.943 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.944 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.945 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.946 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40765/exe 2024-03-25T02:01:33.947 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.948 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.949 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.950 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.951 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.952 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.953 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.954 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_stat 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.955 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_stat 2024-03-25T02:01:33.956 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-25T02:01:33.956 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40931 2024-03-25T02:01:33.956 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_stat on pid 40931' 2024-03-25T02:01:33.956 INFO:tasks.workunit.client.0.smithi083.stdout:test api_stat on pid 40931 2024-03-25T02:01:33.970 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40931 2024-03-25T02:01:33.970 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.970 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.970 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-25T02:01:33.970 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-25T02:01:33.970 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:33.970 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.971 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40781/exe 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.972 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.973 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.974 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:33.975 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.976 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.977 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_stat_pp 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_stat_pp' 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40806/exe 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_stat_pp 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:33.978 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:33.979 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:33.980 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:33.981 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_stat_pp 2024-03-25T02:01:33.982 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40972 2024-03-25T02:01:33.983 INFO:tasks.workunit.client.0.smithi083.stdout:test api_stat_pp on pid 40972 2024-03-25T02:01:33.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_stat_pp on pid 40972' 2024-03-25T02:01:33.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=40972 2024-03-25T02:01:33.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:33.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:33.994 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_watch_notify 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_watch_notify' 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.995 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.996 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.996 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_watch_notify 2024-03-25T02:01:33.996 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_pool: /' 2024-03-25T02:01:33.996 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-25T02:01:33.996 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-25T02:01:33.996 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:33.996 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:33.997 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:33.997 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:33.997 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:33.997 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:33.997 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:33.997 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.997 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:33.998 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.998 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:33.998 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.998 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:33.998 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:33.998 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:33.998 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:33.999 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:33.999 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:33.999 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:33.999 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:33.999 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:33.999 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:33.999 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.000 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.001 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.002 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40882/exe 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.003 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-25T02:01:34.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] Global test environment set-up. 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.836+0000 7fcc1cc37780 10 monclient: get_monmap_and_config 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.836+0000 7fcc1cc37780 10 monclient: build_initial_monmap 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.836+0000 7fcc1cc37780 10 monclient: monmap: 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: epoch 0 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: last_changed 0.000000 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: created 0.000000 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: min_mon_release 0 (unknown) 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: election_strategy: 1 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:01:34.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.837+0000 7fcc1cc37780 1 Processor -- start 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.838+0000 7fcc1cc37780 1 -- start start 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.838+0000 7fcc1cc37780 10 monclient: init 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient: _reopen_session rank -1 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa7490c580 0x55fa749da8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): picked mon.noname-c con 0x55fa7490c580 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa7494b680 0x55fa7490f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): picked mon.noname-b con 0x55fa7494b680 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa749dbb20 0x55fa74946390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): picked mon.noname-a con 0x55fa749dbb20 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): start opening mon connection 2024-03-25T02:01:34.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fa7490d810 con 0x55fa749dbb20 2024-03-25T02:01:34.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): start opening mon connection 2024-03-25T02:01:34.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa748bc3b0 con 0x55fa7494b680 2024-03-25T02:01:34.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): start opening mon connection 2024-03-25T02:01:34.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x55fa749d7360 con 0x55fa7490c580 2024-03-25T02:01:34.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): _renew_subs 2024-03-25T02:01:34.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.839+0000 7fcc1cc37780 10 monclient(hunting): authenticate will time out at 1413.028852s 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.010 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_watch_notify 2024-03-25T02:01:34.011 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41028 2024-03-25T02:01:34.012 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_watch_notify on pid 41028' 2024-03-25T02:01:34.012 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-2test api_watch_notify on pid 41028 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41028 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_watch_notify_pp 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_watch_notify_pp' 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.021 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.022 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.023 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_watch_notify_pp 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.024 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.025 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40931/exe 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.026 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.027 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.028 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_watch_notify_pp 2024-03-25T02:01:34.029 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41052 2024-03-25T02:01:34.030 INFO:tasks.workunit.client.0.smithi083.stdout:test api_watch_notify_pp on pid 41052 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_watch_notify_pp on pid 41052' 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41052 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_cmd 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_cmd' 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-25T02:01:34.043 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.044 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40863/exe 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.045 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.046 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_stat: /' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.047 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-25T02:01:34.048 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.049 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_cmd 2024-03-25T02:01:34.050 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.051 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.051 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.051 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_cmd 2024-03-25T02:01:34.051 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41086 2024-03-25T02:01:34.051 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_cmd on pid 41086' 2024-03-25T02:01:34.051 INFO:tasks.workunit.client.0.smithi083.stdout:test api_cmd on pid 41086 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41086 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stdout:5T02:01:33.847+0000 7fcc1a001640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa7490c580 0x55fa749da8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.853+0000 7fcc1a001640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa7490c580 0x55fa749da8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:34712/0 (socket says 172.21.15.83:34712) 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.853+0000 7fcc1a001640 1 -- 172.21.15.83:0/968815644 learned_addr learned my addr 172.21.15.83:0/968815644 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.853+0000 7fcc1a001640 10 monclient(hunting): get_auth_request con 0x55fa7490c580 auth_method 0 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.853+0000 7fcc1a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:01:34.052 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.853+0000 7fcc1a001640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.853+0000 7fcc1a001640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.863+0000 7fcc1a001640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.863+0000 7fcc1a001640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.863+0000 7fcc1a001640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 1 -- 172.21.15.83:0/968815644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa7494b680 msgr2=0x55fa7490f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 1 --2- 172.21.15.83:0/968815644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa7494b680 0x55fa7490f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 1 -- 172.21.15.83:0/968815644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa749dbb20 msgr2=0x55fa74946390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 1 --2- 172.21.15.83:0/968815644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa749dbb20 0x55fa74946390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 10 monclient: _finish_hunting 0 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 1 monclient: found mon.noname-c 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:01:34.053 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.865+0000 7fcc1a001640 1 -- 172.21.15.83:0/968815644 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa74841ca0 con 0x55fa7490c580 2024-03-25T02:01:34.054 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.869+0000 7fcc1a001640 1 --2- 172.21.15.83:0/968815644 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa7490c580 0x55fa749da8a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc140016a0 tx=0x7fcc14001b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=245fba1d7fe98da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:01:34.058 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-25T02:01:34.058 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.058 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.058 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.058 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.059 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_cmd_pp 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_cmd_pp' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_cmd_pp 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.060 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_cmd_pp 2024-03-25T02:01:34.061 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41110 2024-03-25T02:01:34.062 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc0bfff640 1 -- 172.21.15.83:0/9688test api_cmd_pp on pid 41110 2024-03-25T02:01:34.068 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_cmd_pp on pid 41110' 2024-03-25T02:01:34.068 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41110 2024-03-25T02:01:34.068 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.068 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.068 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.068 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.068 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/40972/exe 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_service 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_service' 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.069 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.070 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_service 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.071 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.072 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_service 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41133 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_service on pid 41133' 2024-03-25T02:01:34.073 INFO:tasks.workunit.client.0.smithi083.stdout:test api_service on pid 41133 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41133 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_service_pp 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_service_pp' 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.087 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.088 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_service_pp 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.089 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.090 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.091 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.092 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.093 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.094 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.095 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.096 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.097 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.098 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.099 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.099 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.099 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.099 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_service_pp 2024-03-25T02:01:34.099 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41173 2024-03-25T02:01:34.099 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_service_pp on pid 41173' 2024-03-25T02:01:34.099 INFO:tasks.workunit.client.0.smithi083.stdout:test api_service_pp on pid 41173 2024-03-25T02:01:34.105 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41173 2024-03-25T02:01:34.105 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.105 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.105 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.105 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.105 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_c_write_operations 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_c_write_operations' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.106 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_c_write_operations 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.107 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.108 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41028/exe 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.109 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_c_write_operations 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.110 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41052/exe 2024-03-25T02:01:34.111 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.111 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.111 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.111 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41186 2024-03-25T02:01:34.111 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.111 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.111 INFO:tasks.workunit.client.0.smithi083.stdout:test api_c_write_operations on pid 41186 2024-03-25T02:01:34.126 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.126 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_c_write_operations on pid 41186' 2024-03-25T02:01:34.126 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.126 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41186 2024-03-25T02:01:34.126 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.126 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.127 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_c_read_operations 2024-03-25T02:01:34.128 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_c_read_operations' 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.129 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.130 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41086/exe 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.131 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.132 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.133 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.134 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.135 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.136 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.137 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_c_read_operations 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.138 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-25T02:01:34.139 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_c_read_operations 2024-03-25T02:01:34.140 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.140 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-25T02:01:34.140 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41221 2024-03-25T02:01:34.140 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_c_read_operations on pid 41221' 2024-03-25T02:01:34.140 INFO:tasks.workunit.client.0.smithi083.stdout:test api_c_read_operations on pid 41221 2024-03-25T02:01:34.148 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41221 2024-03-25T02:01:34.148 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.148 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.148 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s api_cls_remote_reads 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' api_cls_remote_reads' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo api_cls_remote_reads 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.149 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.150 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.151 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.151 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.151 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.151 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.151 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.151 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.152 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.153 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.153 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.153 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=api_cls_remote_reads 2024-03-25T02:01:34.153 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41233 2024-03-25T02:01:34.153 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test api_cls_remote_reads on pid 41233' 2024-03-25T02:01:34.154 INFO:tasks.workunit.client.0.smithi083.stdout:test api_cls_remote_reads on pid 41233 2024-03-25T02:01:34.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-25T02:01:34.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41233 2024-03-25T02:01:34.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.159 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s list_parallel 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' list_parallel' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_cmd: /' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo list_parallel 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41110/exe 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.160 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.161 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.162 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.162 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.162 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=list_parallel 2024-03-25T02:01:34.162 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.162 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.162 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.163 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.163 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41243 2024-03-25T02:01:34.163 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.163 INFO:tasks.workunit.client.0.smithi083.stdout:test list_parallel on pid 41243 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test list_parallel on pid 41243' 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41243 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.173 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.174 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.175 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.176 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s open_pools_parallel 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' open_pools_parallel' 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.177 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.178 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41133/exe 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.179 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo open_pools_parallel 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout:15644 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14016070 con 0x55fa7490c580 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc0bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc0bfff640 10 monclient: dump: 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: epoch 1 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: min_mon_release 19 (squid) 2024-03-25T02:01:34.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: election_strategy: 1 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc0bfff640 10 monclient: _finish_auth 0 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc0bfff640 10 monclient: _check_auth_tickets 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc1cc37780 5 monclient: authenticate success, global_id 4349 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.888+0000 7fcc1cc37780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:01:34.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.954+0000 7fcc0bfff640 1 -- 172.21.15.83:0/968815644 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc14006230 con 0x55fa7490c580 2024-03-25T02:01:34.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.954+0000 7fcc0bfff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:01:34.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.955+0000 7fcc1cc37780 10 monclient: get_monmap_and_config success 2024-03-25T02:01:34.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.955+0000 7fcc1cc37780 4 set_mon_vals no callback set 2024-03-25T02:01:34.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.955+0000 7fcc1cc37780 10 monclient: shutdown 2024-03-25T02:01:34.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.955+0000 7fcc1cc37780 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:01:34.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.955+0000 7fcc1cc37780 1 -- 172.21.15.83:0/968815644 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa7490c580 msgr2=0x55fa749da8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:01:34.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.955+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/968815644 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa7490c580 0x55fa749da8a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc140016a0 tx=0x7fcc14001b60 comp rx=0 tx=0).stop 2024-03-25T02:01:34.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.954+0000 7fcc0bfff640 1 -- 172.21.15.83:0/968815644 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc140107d0 con 0x55fa7490c580 2024-03-25T02:01:34.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.955+0000 7fcc0bfff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-03-25T02:01:34.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 -- 172.21.15.83:0/968815644 shutdown_connections 2024-03-25T02:01:34.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/968815644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa749dbb20 0x55fa74946390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:01:34.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/968815644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa7494b680 0x55fa7490f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/968815644 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa7490c580 0x55fa749da8a0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc140016a0 tx=0x7fcc14001b60 comp rx=0 tx=0).stop 2024-03-25T02:01:34.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 -- 172.21.15.83:0/968815644 >> 172.21.15.83:0/968815644 conn(0x55fa74908520 msgr2=0x55fa749d9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 -- 172.21.15.83:0/968815644 shutdown_connections 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 -- 172.21.15.83:0/968815644 wait complete. 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 10 monclient: build_initial_monmap 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 10 monclient: monmap: 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: epoch 0 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: last_changed 0.000000 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: created 0.000000 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: min_mon_release 0 (unknown) 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: election_strategy: 1 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:01:34.186 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 librados: starting msgr at 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 librados: starting objecter 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.956+0000 7fcc1cc37780 1 Processor -- start 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 -- start start 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 librados: setting wanted keys 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 librados: calling monclient init 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient: init 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient: _reopen_session rank -1 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:01:34.187 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa7494b680 0x55fa749e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): picked mon.noname-a con 0x55fa7494b680 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa749dbb20 0x55fa749e01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): picked mon.noname-b con 0x55fa749dbb20 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa749df7c0 0x55fa749cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): picked mon.noname-c con 0x55fa749df7c0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): start opening mon connection 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fa749d7030 con 0x55fa7494b680 2024-03-25T02:01:34.188 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): start opening mon connection 2024-03-25T02:01:34.193 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.193 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.193 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.193 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.193 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.193 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.194 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.195 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_service: /' 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_service.log 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-25T02:01:34.196 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.197 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=open_pools_parallel 2024-03-25T02:01:34.197 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41286 2024-03-25T02:01:34.197 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957test open_pools_parallel on pid 41286 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test open_pools_parallel on pid 41286' 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41286 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.215 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.216 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s delete_pools_parallel 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' delete_pools_parallel' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.217 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.218 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.219 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.220 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.221 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.222 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.223 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.224 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.225 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41173/exe 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41186/exe 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.226 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.227 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo delete_pools_parallel 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.228 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stdout:+0000 7fcc1cc37780 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55fa749d71b0 con 0x55fa749dbb20 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): start opening mon connection 2024-03-25T02:01:34.229 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x55fa7494d640 con 0x55fa749df7c0 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): _renew_subs 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.957+0000 7fcc1cc37780 10 monclient(hunting): authenticate will time out at 1413.146813s 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.979+0000 7fcc1a001640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa7494b680 0x55fa749e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.979+0000 7fcc1a001640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa7494b680 0x55fa749e1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:51880/0 (socket says 172.21.15.83:51880) 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.979+0000 7fcc1a001640 1 -- 172.21.15.83:0/2311987957 learned_addr learned my addr 172.21.15.83:0/2311987957 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.979+0000 7fcc1a001640 10 monclient(hunting): get_auth_request con 0x55fa7494b680 auth_method 0 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.979+0000 7fcc1a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.979+0000 7fcc1a001640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.979+0000 7fcc1a001640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.981+0000 7fcc1a001640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.981+0000 7fcc1a001640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.981+0000 7fcc1a001640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:01:34.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.985+0000 7fcc1a802640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa749df7c0 0x55fa749cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:01:34.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.985+0000 7fcc1a802640 10 monclient(hunting): get_auth_request con 0x55fa749df7c0 auth_method 0 2024-03-25T02:01:34.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.985+0000 7fcc1a802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:01:34.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.985+0000 7fcc1a802640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:01:34.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.985+0000 7fcc1a802640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:01:34.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2024-03-25T02:01:34.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 1 -- 172.21.15.83:0/2311987957 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa749df7c0 msgr2=0x55fa749cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:01:34.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x55fa749df7c0 0x55fa749cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:01:34.233 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-25T02:01:34.233 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-25T02:01:34.233 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-25T02:01:34.233 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=delete_pools_parallel 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.234 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41331 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.235 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test delete_pools_parallel on pid 41331' 2024-03-25T02:01:34.236 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 1 -- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:3300/0,v1:172.test delete_pools_parallel on pid 41331 2024-03-25T02:01:34.252 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.252 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.252 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.252 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41331 2024-03-25T02:01:34.253 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.254 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s cls 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' cls' 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.255 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.256 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.257 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.258 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.259 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.260 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41221/exe 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.261 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.262 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.263 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.264 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.265 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.266 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo cls 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=cls 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.267 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.268 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41380 2024-03-25T02:01:34.268 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test cls on pid 41380' 2024-03-25T02:01:34.268 INFO:tasks.workunit.client.0.smithi083.stdout:test cls on pid 41380 2024-03-25T02:01:34.270 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41380 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-25T02:01:34.271 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stdout:21.15.153:6789/0] conn(0x55fa749dbb20 msgr2=0x55fa749e01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55fa749dbb20 0x55fa749e01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 10 monclient: _finish_hunting 0 2024-03-25T02:01:34.272 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 1 monclient: found mon.noname-a 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.987+0000 7fcc1a001640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa749e1140 con 0x55fa7494b680 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.997+0000 7fcc1a001640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa7494b680 0x55fa749e1e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc14006480 tx=0x7fcc14001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.997+0000 7fcc0affd640 10 client.?.objecter ms_handle_connect 0x55fa7494b680 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:33.997+0000 7fcc0affd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14020030 con 0x55fa7494b680 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: dump: 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: epoch 1 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:01:34.273 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: min_mon_release 19 (squid) 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: election_strategy: 1 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: _finish_auth 0 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: _check_auth_tickets 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc14006db0 con 0x55fa7494b680 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14016040 con 0x55fa7494b680 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:01:34.274 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:01:34.275 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.008+0000 7fcc0affd640 10 monclient: dump: 2024-03-25T02:01:34.282 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.282 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.282 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.282 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s cmd 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' cmd' 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.283 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo cmd 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.284 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.285 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41233/exe 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.286 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.287 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=cmd 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.288 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.289 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41441 2024-03-25T02:01:34.289 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test cmd on pid 41441' 2024-03-25T02:01:34.289 INFO:tasks.workunit.client.0.smithi083.stdout: test cmd on pid 41441 2024-03-25T02:01:34.299 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41441 2024-03-25T02:01:34.299 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.299 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.299 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41243/exe 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s handler_error 2024-03-25T02:01:34.300 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' handler_error' 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.301 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.302 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.303 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-25T02:01:34.304 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ list_parallel: /' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.305 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo handler_error 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.306 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=handler_error 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41465 2024-03-25T02:01:34.307 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test handler_error on pid 41465' 2024-03-25T02:01:34.308 INFO:tasks.workunit.client.0.smithi083.stdout:test handler_error on pid 41465 2024-03-25T02:01:34.317 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.317 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.317 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41465 2024-03-25T02:01:34.317 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.317 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.317 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.318 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s io 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' io' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo io 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.319 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.320 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.321 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.322 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.323 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.324 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.325 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.325 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.325 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=io 2024-03-25T02:01:34.325 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41501 2024-03-25T02:01:34.325 INFO:tasks.workunit.client.0.smithi083.stdout:test io on pid 41501 2024-03-25T02:01:34.329 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test io on pid 41501' 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41501 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s ec_io 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' ec_io' 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41286/exe 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo ec_io 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.330 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.331 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.332 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=ec_io 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.333 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.334 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.334 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41513 2024-03-25T02:01:34.334 INFO:tasks.workunit.client.0.smithi083.stdout:test ec_io on pid 41513 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test ec_io on pid 41513' 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41513 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.343 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-25T02:01:34.344 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s list 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' list' 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.345 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.346 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.346 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.346 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.346 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.346 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.346 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.346 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.347 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.347 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.347 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.347 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.347 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.347 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-25T02:01:34.347 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.348 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41380/exe 2024-03-25T02:01:34.349 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41331/exe 2024-03-25T02:01:34.349 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo list 2024-03-25T02:01:34.349 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.349 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.349 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.349 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.350 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.350 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.350 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.350 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.350 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.350 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.350 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.351 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.351 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.351 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.351 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.351 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.351 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.352 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.352 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.352 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.352 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.352 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.352 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.352 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.353 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.353 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.353 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.353 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.353 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.353 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.353 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.354 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.354 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.354 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.354 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.354 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.354 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.355 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.355 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ cls: /' 2024-03-25T02:01:34.355 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.355 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.355 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.355 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-25T02:01:34.355 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-25T02:01:34.356 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-25T02:01:34.356 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=list 2024-03-25T02:01:34.356 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41544 2024-03-25T02:01:34.356 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test list on pid 41544' 2024-03-25T02:01:34.357 INFO:tasks.workunit.client.0.smithi083.stdout:test list on pid 41544 2024-03-25T02:01:34.380 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41544 2024-03-25T02:01:34.380 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.380 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.380 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-25T02:01:34.380 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.381 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.382 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.383 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.384 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s ec_list 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' ec_list' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.385 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_cls.log 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_cls 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.386 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.387 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.388 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.389 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.390 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.391 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo ec_list 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=ec_list 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.392 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41601 2024-03-25T02:01:34.393 INFO:tasks.workunit.client.0.smithi083.stdout:test ec_list on pid 41601 2024-03-25T02:01:34.404 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test ec_list on pid 41601' 2024-03-25T02:01:34.404 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41601 2024-03-25T02:01:34.404 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.405 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41441/exe 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.406 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.407 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.408 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41513/exe 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ cmd: /' 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s misc 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.409 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.410 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' misc' 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_cmd 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.411 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ ec_io: /' 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.412 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.413 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo misc 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: epoch 1 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: min_mon_release 19 (squid) 2024-03-25T02:01:34.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: election_strategy: 1 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 5 monclient: authenticate success, global_id 4422 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 10 monclient: _renew_subs 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fa749e1420 con 0x55fa7494b680 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 10 monclient: _renew_subs 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa749dfcc0 con 0x55fa7494b680 2024-03-25T02:01:34.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 1 librados: init done 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc1cc37780 10 librados: wait_for_osdmap waiting 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.009+0000 7fcc0bfff640 4 set_mon_vals no callback set 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.121+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fcc14023030 con 0x55fa7494b680 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.122+0000 7fcc0affd640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fcbfc06bce0 0x7fcbfc06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.122+0000 7fcc19800640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fcbfc06bce0 0x7fcbfc06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.122+0000 7fcc19800640 10 monclient: get_auth_request con 0x7fcbfc06bce0 auth_method 0 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.123+0000 7fcc19800640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fcbfc06bce0 0x7fcbfc06e1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc10058000 tx=0x7fcc10050070 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.123+0000 7fcc0affd640 10 client.4422.objecter ms_handle_connect 0x7fcbfc06bce0 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fcc14056ab0 con 0x55fa7494b680 2024-03-25T02:01:34.416 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc0affd640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_map(25..25 src has 1..25) 2024-03-25T02:01:34.417 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-25T02:01:34.417 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_ec_io 2024-03-25T02:01:34.417 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map got epochs [25test misc on pid 41646 2024-03-25T02:01:34.427 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=misc 2024-03-25T02:01:34.427 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41646 2024-03-25T02:01:34.427 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test misc on pid 41646' 2024-03-25T02:01:34.427 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41646 2024-03-25T02:01:34.427 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.427 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.427 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.428 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s pool 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' pool' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41465/exe 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.429 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo pool 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.430 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.431 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.432 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.433 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.434 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.435 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.436 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.437 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_handler_error 2024-03-25T02:01:34.437 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.437 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.437 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=pool 2024-03-25T02:01:34.437 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41707 2024-03-25T02:01:34.437 INFO:tasks.workunit.client.0.smithi083.stdout:test pool on pid 41707 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test pool on pid 41707' 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41707 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ handler_error: /' 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.452 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.453 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.454 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.455 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s read_operations 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' read_operations' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.456 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.457 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.458 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41544/exe 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo read_operations 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.459 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.460 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.461 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.462 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_list 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.463 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=read_operations 2024-03-25T02:01:34.464 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41756 2024-03-25T02:01:34.464 INFO:tasks.workunit.client.0.smithi083.stdout:test read_operations on pid 41756 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test read_operations on pid 41756' 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41756 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ list: /' 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s snapshots 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' snapshots' 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.470 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_list.log 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo snapshots 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.471 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.472 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41601/exe 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.473 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=snapshots 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41769 2024-03-25T02:01:34.474 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test snapshots on pid 41769' 2024-03-25T02:01:34.475 INFO:tasks.workunit.client.0.smithi083.stdout:test snapshots on pid 41769 2024-03-25T02:01:34.488 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41769 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s watch_notify 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' watch_notify' 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.489 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.490 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.491 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.492 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_ec_list 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41501/exe 2024-03-25T02:01:34.493 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.494 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.495 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.496 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ ec_list: /' 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.497 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_io 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.498 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_io.log 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ io: /' 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.499 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.500 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo watch_notify 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41646/exe 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.501 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.502 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.502 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.502 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.502 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=watch_notify 2024-03-25T02:01:34.502 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41802 2024-03-25T02:01:34.502 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test watch_notify on pid 41802' 2024-03-25T02:01:34.502 INFO:tasks.workunit.client.0.smithi083.stdout:test watch_notify on pid 41802 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41802 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.512 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ printf %25s write_operations 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:+ r=' write_operations' 2024-03-25T02:01:34.513 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.514 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.515 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.516 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:++ echo write_operations 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.517 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:++ awk '{print $1}' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_misc 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_misc.log 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ misc: /' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.518 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41707/exe 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+ ff=write_operations 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41825 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.519 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.520 INFO:tasks.workunit.client.0.smithi083.stdout:test write_operations on pid 41825 2024-03-25T02:01:34.545 INFO:tasks.workunit.client.0.smithi083.stderr:+ echo 'test write_operations on pid 41825' 2024-03-25T02:01:34.545 INFO:tasks.workunit.client.0.smithi083.stderr:+ pids[$f]=41825 2024-03-25T02:01:34.545 INFO:tasks.workunit.client.0.smithi083.stderr:+ ret=0 2024-03-25T02:01:34.545 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.545 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' 1 -eq 1 ']' 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40781 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40781 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.546 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -f /etc/bashrc ']' 2024-03-25T02:01:34.547 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.547 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.547 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.547 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.547 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.547 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.547 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:+ . /etc/bashrc 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.548 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_pool 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -z '' ']' 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ BASHRCSOURCED=Y 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ shopt -q login_shell 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:+++ umask 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' 0022 -eq 0 ']' 2024-03-25T02:01:34.549 INFO:tasks.workunit.client.0.smithi083.stderr:++ SHELL=/bin/bash 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ pool: /' 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-25T02:01:34.550 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_pool.log 2024-03-25T02:01:34.551 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.553 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.553 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.553 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.553 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.553 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.553 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.553 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.554 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'grep=grep --color=auto' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-25T02:01:34.555 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorls.sh 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' '!' -t 0 ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:+++ return 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.556 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export S_COLORS=auto 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:+++ S_COLORS=auto 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-25T02:01:34.557 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41756/exe 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.558 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_read_operations 2024-03-25T02:01:34.559 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.560 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ read_operations: /' 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41769/exe 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.561 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_snapshots 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ snapshots: /' 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.562 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.563 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/gawk.sh 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.564 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/lang.sh 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/locale 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.565 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.566 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41825/exe 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.567 INFO:tasks.workunit.client.0.smithi083.stderr:++++ export LANG=en_US.UTF-8 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:++++ LANG=en_US.UTF-8 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ LANG=en_US.UTF-8 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ unset LANG_backup config 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n '' ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' dumb = linux ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/less.sh 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-25T02:01:34.568 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:++ for i in /etc/profile.d/*.sh 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:++ '[' '' ']' 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:++ . /etc/profile.d/which2.sh 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+++++ readlink /proc/41802/exe 2024-03-25T02:01:34.569 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:++++ basename /usr/bin/bash 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ write_operations: /' 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_write_operations 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_declare='declare -f' 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+++ which_opt=-f 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export which_declare 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+++ export -f which 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset i 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:++ unset -f pathmunge 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+ export PATH 2024-03-25T02:01:34.570 INFO:tasks.workunit.client.0.smithi083.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-25T02:01:34.571 INFO:tasks.workunit.client.0.smithi083.stderr:+ unset rc 2024-03-25T02:01:34.571 INFO:tasks.workunit.client.0.smithi083.stderr:+ ceph_test_neorados_watch_notify 2024-03-25T02:01:34.571 INFO:tasks.workunit.client.0.smithi083.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-25T02:01:34.571 INFO:tasks.workunit.client.0.smithi083.stderr:+ sed 's/^/ watch_notify: /' 2024-03-25T02:01:35.157 INFO:tasks.workunit.client.0.smithi083.stdout:,25] > 0 2024-03-25T02:01:35.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding full epoch 25 2024-03-25T02:01:35.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc0affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T02:01:35.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc1cc37780 10 librados: wait_for_osdmap done waiting 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc1cc37780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.129+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc14006230 con 0x55fa7494b680 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.166+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcc1402a070 con 0x55fa7494b680 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.166+0000 7fcc0affd640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.166+0000 7fcc0b7fe640 10 client.4422.objecter _get_latest_version latest 25, have it 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.166+0000 7fcc1cc37780 10 client.4422.objecter create_pool name=ceph_test_rados_api_asio 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.166+0000 7fcc1cc37780 10 client.4422.objecter pool_op_submit 1 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.166+0000 7fcc1cc37780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:34.166+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x55fa748c2d50 con 0x55fa7494b680 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v26) ==== 43+0+0 (secure 0 0 0) 0x7fcc14059110 con 0x55fa7494b680 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 pool_op_reply(tid 1 (0) Success v26) 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 10 client.4422.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v26) 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 10 client.4422.objecter have request 1 at 0x55fa749dffa0 Op: create 2024-03-25T02:01:35.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 20 client.4422.objecter waiting for client to reach epoch 26 before calling back 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 10 monclient: _renew_subs 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=26}) -- 0x7fcbfc0719c0 con 0x55fa7494b680 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.090+0000 7fcc0affd640 10 client.4422.objecter done 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 8 ==== osd_map(26..26 src has 1..26) ==== 3969+0+0 (secure 0 0 0) 0x7fcc14056d10 con 0x55fa7494b680 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc0affd640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_map(26..26 src has 1..26) 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map got epochs [26,26] > 25 2024-03-25T02:01:35.159 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 26 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc0affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 10 librados: set snap read head -> 1 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 10 librados: writefull oid=exist nspace= 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749d5f20 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 26 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x55fa749d63d0 0x55fa74a7ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 2 osd.7 2024-03-25T02:01:37.959 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 2 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 15 client.4422.objecter _send_op 2 to 9.1 on osd.7 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:2 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) -- 0x55fa74a7b1d0 con 0x55fa749d63d0 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.156+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.157+0000 7fcc1a802640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x55fa749d63d0 0x55fa74a7ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.157+0000 7fcc1a802640 10 monclient: get_auth_request con 0x55fa749d63d0 auth_method 0 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.160+0000 7fcc1a802640 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x55fa749d63d0 0x55fa74a7ac90 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:35.160+0000 7fcc0affd640 10 client.4422.objecter ms_handle_connect 0x55fa749d63d0 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:36.957+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:36.957+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:36.957+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:36.957+0000 7fcc09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:01:37.960 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:37.957+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:37.961 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:37.957+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:45.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:37.957+0000 7fcc09ffb640 20 api_service: [==========] Running 4 tests from 1 test suite. 2024-03-25T02:01:45.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] Global test environment set-up. 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5253 ms) 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.RegisterLate (59 ms) 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 20 pool(s) do not have an application enabled 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-03-25T02:01:45.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 76s), 8 in (since 2m) 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 35 pools, 1065 pgs 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 2 objects, 449 KiB 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 95.305% pgs unknown 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 3.850% pgs not active 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 1015 unknown 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 41 creating+peering 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 9 active+clean 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.222 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 20 pool(s) do not have an application enabled 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 79s), 8 in (since 2m) 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: foo: 10 portals active (1 hosts, 3 zones) 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 40 pools, 1153 pgs 2024-03-25T02:01:45.223 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 2 objects, 449 KiB 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 69.211% pgs unknown 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 28.274% pgs not active 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 798 unknown 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 280 creating+peering 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 46 creating+activating 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 29 active+clean 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: cluster: 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: id: 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: health: HEALTH_WARN 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 20 pool(s) do not have an application enabled 2024-03-25T02:01:45.224 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: services: 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: osd: 8 osds: 8 up (since 81s), 8 in (since 2m) 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: data: 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pools: 40 pools, 1153 pgs 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: objects: 3 objects, 449 KiB 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-03-25T02:01:45.225 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: pgs: 39.896% pgs unknown 2024-03-25T02:01:47.807 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:30.642 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:47.857 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:30.692 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:47.859 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:30.651 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:47.865 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:30.657 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:47.875 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:30.710 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:47.894 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:30.728 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:48.068 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:02:30.904 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:48.083 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:02:30.875 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:48.093 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:02:30.928 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:38.958+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:38.958+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:38.958+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:39.010+0000 7fcc1c265640 10 client.4422.objecter tick 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:39.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:39.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:39.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:40.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:40.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:40.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.013 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:41.958+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:41.958+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:41.958+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:42.958+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:42.958+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:42.958+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:43.958+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:43.958+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:43.958+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:44.010+0000 7fcc1c265640 10 client.4422.objecter tick 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:44.958+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:44.958+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:44.958+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:45.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.014 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:45.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:45.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:46.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:46.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:46.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:46.959+0000 7fcc09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:47.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:47.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:47.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:48.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:48.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:48.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:49.015 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.011+0000 7fcc1c265640 10 client.4422.objecter tick 2024-03-25T02:01:49.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.011+0000 7fcc1c265640 2 client.4422.objecter tid 2 on osd.7 is laggy 2024-03-25T02:01:49.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.011+0000 7fcc1c265640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:01:49.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.011+0000 7fcc1c265640 10 monclient: _renew_subs 2024-03-25T02:01:49.462 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.011+0000 7fcc1c265640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.011+0000 7fcc1c265640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=27}) -- 0x7fcbec002f90 con 0x55fa7494b680 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.011+0000 7fcc1c265640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- ping magic: 0 -- 0x7fcbec003360 con 0x55fa749d63d0 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 9 ==== osd_map(27..35 src has 1..35) ==== 32104+0+0 (secure 0 0 0) 0x7fcc14055080 con 0x55fa7494b680 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_map(27..35 src has 1..35) 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map got epochs [27,35] > 26 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 27 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 27 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 28 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 28 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:49.463 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 29 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 29 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 30 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 30 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 31 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:49.464 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 31 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:53.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:53.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 32 2024-03-25T02:01:53.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:53.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 32 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:53.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:53.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 33 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 33 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 34 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 34 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 35 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 35 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.460+0000 7fcc0affd640 20 client.4422.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-03-25T02:01:53.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:49.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:50.959+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:50.959+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:50.959+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:51.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:51.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:51.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:52.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:52.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:52.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:53.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:53.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:53.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:57.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:53.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:57.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.012+0000 7fcc1c265640 10 client.4422.objecter tick 2024-03-25T02:01:57.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.012+0000 7fcc1c265640 2 client.4422.objecter tid 2 on osd.7 is laggy 2024-03-25T02:01:57.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.012+0000 7fcc1c265640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:01:57.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.012+0000 7fcc1c265640 10 monclient: _renew_subs 2024-03-25T02:01:57.962 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.012+0000 7fcc1c265640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.012+0000 7fcc1c265640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=36}) -- 0x7fcbec003560 con 0x55fa7494b680 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.012+0000 7fcc1c265640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- ping magic: 0 -- 0x7fcbec003790 con 0x55fa749d63d0 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 10 ==== osd_map(36..37 src has 1..37) ==== 1696+0+0 (secure 0 0 0) 0x7fcc14058c70 con 0x55fa7494b680 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_map(36..37 src has 1..37) 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map got epochs [36,37] > 35 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 36 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 36 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:57.963 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 37 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 37 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.033+0000 7fcc0affd640 20 client.4422.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:54.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:55.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:55.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:55.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:56.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:01:57.964 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:56.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:01:57.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:56.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:01:57.965 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:56.960+0000 7fcc09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:02:00.096 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:57.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:57.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:57.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.760+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 1 ==== osd_map(27..38 src has 1..38) ==== 34056+0+0 (crc 0 0 0) 0x7fcc0c0097d0 con 0x55fa749d63d0 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.760+0000 7fcc0affd640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_map(27..38 src has 1..38) 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.761+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map got epochs [27,38] > 37 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.761+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 38 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.761+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.761+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 38 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:00.097 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.761+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.761+0000 7fcc0affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.761+0000 7fcc0affd640 20 client.4422.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:58.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.013+0000 7fcc1c265640 10 client.4422.objecter tick 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.013+0000 7fcc1c265640 2 client.4422.objecter tid 2 on osd.7 is laggy 2024-03-25T02:02:00.098 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.013+0000 7fcc1c265640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.013+0000 7fcc1c265640 10 monclient: _renew_subs 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.013+0000 7fcc1c265640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.013+0000 7fcc1c265640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=39}) -- 0x7fcbec003560 con 0x55fa7494b680 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.013+0000 7fcc1c265640 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- ping magic: 0 -- 0x7fcbec003a50 con 0x55fa749d63d0 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:01:59.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.094+0000 7fcc0affd640 1 -- 172.21.15.83:0/2311987957 <== mon.0 v2:172.21.15.83:3300/0 11 ==== osd_map(39..39 src has 1..39) ==== 692+0+0 (secure 0 0 0) 0x7fcc14058e70 con 0x55fa7494b680 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_map(39..39 src has 1..39) 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map got epochs [39,39] > 38 2024-03-25T02:02:00.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 3 client.4422.objecter handle_osd_map decoding incremental epoch 39 2024-03-25T02:02:00.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 10 client.4422.objecter checking op 2 2024-03-25T02:02:01.344 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.095+0000 7fcc0affd640 20 client.4422.objecter 2 9.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:00.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 2 ==== osd_op_reply(2 exist [writefull 0~5] v38'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(2 exist [writefull 0~5] v38'1 uv1 ondisk = 0) 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 2 ondisk uv 1 in 9.1 attempt 0 2024-03-25T02:02:01.345 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 2 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 15 client.4422.objecter _finish_op 2 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 2 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 10 librados: Objecter returned from writefull r=0 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749d5f20 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 3 osd.7 2024-03-25T02:02:01.346 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:01.347 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 3 2024-03-25T02:02:01.347 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 15 client.4422.objecter _send_op 3 to 9.1 on osd.7 2024-03-25T02:02:01.347 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:3 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa749cf8d0 con 0x55fa749d63d0 2024-03-25T02:02:01.494 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa handler_error: Running main() from gmock_main.cc 2024-03-25T02:02:01.494 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-25T02:02:01.494 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [----------] Global test environment set-up. 2024-03-25T02:02:01.494 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-25T02:02:01.494 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-25T02:02:01.494 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (27318 ms) 2024-03-25T02:02:01.494 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [----------] 1 test from neocls_handler_error (27318 ms total) 2024-03-25T02:02:01.495 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: 2024-03-25T02:02:01.495 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [----------] Global test environment tear-down 2024-03-25T02:02:01.495 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [==========] 1 test from 1 test suite ran. (27318 ms total) 2024-03-25T02:02:01.495 INFO:tasks.workunit.client.0.smithi083.stdout: handler_error: [ PASSED ] 1 test. 2024-03-25T02:02:03.181 INFO:tasks.workunit.client.0.smithi083.stdout:749cc890 osd=7 4 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749cfca0 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 4 osd.7 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 4 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 15 client.4422.objecter _send_op 4 to 9.1 on osd.7 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:4 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa749d0150 con 0x55fa749d63d0 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.343+0000 7fcc1cc37780 5 client.4422.objecter 2 in flight 2024-03-25T02:02:03.182 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.960+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.960+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:01.960+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:02.961+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:02.961+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:02.961+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 3 ondisk uv 1 in 9.1 attempt 0 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 5 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 3 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 15 client.4422.objecter _finish_op 3 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:03.183 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 3 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 5 client.4422.objecter 1 in flight 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 4 ondisk uv 0 in 9.1 attempt 0 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 10 client.4422.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 4 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 15 client.4422.objecter _finish_op 4 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 4 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.179+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1837 ms) 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-25T02:02:03.184 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749cfca0 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 5 osd.7 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 5 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 15 client.4422.objecter _send_op 5 to 9.1 on osd.7 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:5 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa749d04f0 con 0x55fa749d63d0 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749d5f20 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-03-25T02:02:03.185 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:03.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 6 osd.7 2024-03-25T02:02:03.624 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 6 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 15 client.4422.objecter _send_op 6 to 9.1 on osd.7 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:6 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa749c8d20 con 0x55fa749d63d0 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.180+0000 7fcc1cc37780 5 client.4422.objecter 2 in flight 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 5 ondisk uv 1 in 9.1 attempt 0 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 5 2024-03-25T02:02:03.625 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 5 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 15 client.4422.objecter _finish_op 5 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 5 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 5 client.4422.objecter 1 in flight 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 6 ondisk uv 0 in 9.1 attempt 0 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 10 client.4422.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:03.626 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 6 2024-03-25T02:02:03.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 15 client.4422.objecter _finish_op 6 2024-03-25T02:02:03.627 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.842 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 6 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.622+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (443 ms) 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749d5f20 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 7 osd.7 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 7 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 15 client.4422.objecter _send_op 7 to 9.1 on osd.7 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:7 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa749cfca0 con 0x55fa749d63d0 2024-03-25T02:02:03.843 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749c98d0 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 8 osd.7 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 8 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 15 client.4422.objecter _send_op 8 to 9.1 on osd.7 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:8 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa74ad07b0 con 0x55fa749d63d0 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.623+0000 7fcc1cc37780 5 client.4422.objecter 2 in flight 2024-03-25T02:02:03.844 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 7 ondisk uv 1 in 9.1 attempt 0 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (218 ms) 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 5 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 7 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 15 client.4422.objecter _finish_op 7 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 7 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 5 client.4422.objecter 1 in flight 2024-03-25T02:02:03.845 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 8 ondisk uv 0 in 9.1 attempt 0 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 10 client.4422.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 8 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 15 client.4422.objecter _finish_op 8 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 8 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.840+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 librados: queue_aio_write 0x55fa7487ec30 completion 0x55fa7485a810 write_seq 1 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749d5f20 2024-03-25T02:02:03.846 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:03.847 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:03.847 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:03.847 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:04.886 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 9 osd.7 2024-03-25T02:02:04.886 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:04.886 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 9 2024-03-25T02:02:04.886 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 15 client.4422.objecter _send_op 9 to 9.1 on osd.7 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:9 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) -- 0x55fa749c98d0 con 0x55fa749d63d0 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.841+0000 7fcc1cc37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.961+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.961+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:03.961+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.014+0000 7fcc1c265640 10 client.4422.objecter tick 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 9 ==== osd_op_reply(9 exist [write 0~5] v39'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(9 exist [write 0~5] v39'2 uv2 ondisk = 0) 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 9 ondisk uv 2 in 9.1 attempt 0 2024-03-25T02:02:04.887 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 9 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 15 client.4422.objecter _finish_op 9 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 9 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (1044 ms) 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1a802640 20 librados: complete_aio_write 0x55fa7485a810 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 librados: queue_aio_write 0x55fa7487ec30 completion 0x55fa74849b40 write_seq 2 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa749d5f20 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:04.888 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:04.889 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 10 osd.7 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 10 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 15 client.4422.objecter _send_op 10 to 9.1 on osd.7 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:10 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) -- 0x55fa749c98d0 con 0x55fa749d63d0 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.884+0000 7fcc1cc37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.961+0000 7fcc09ffb640 10 monclient: tick 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.961+0000 7fcc09ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:04.961+0000 7fcc09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 10 ==== osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(10 exist [write 0~5] v39'3 uv3 ondisk = 0) 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 10 ondisk uv 3 in 9.1 attempt 0 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 10 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 15 client.4422.objecter _finish_op 10 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 10 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1a802640 20 librados: complete_aio_write 0x55fa74849b40 2024-03-25T02:02:05.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (249 ms) 2024-03-25T02:02:05.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-25T02:02:05.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-03-25T02:02:05.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 librados: queue_aio_write 0x55fa7487ec30 completion 0x55fa749d5f20 write_seq 3 2024-03-25T02:02:05.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74ab34a0 2024-03-25T02:02:05.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:05.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:05.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 11 osd.7 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 11 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 15 client.4422.objecter _send_op 11 to 9.1 on osd.7 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:11 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) -- 0x55fa74ab3950 con 0x55fa749d63d0 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.133+0000 7fcc1cc37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 11 ==== osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(11 exist [write 0~5] v39'4 uv4 ondisk = 0) 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 11 ondisk uv 4 in 9.1 attempt 0 2024-03-25T02:02:05.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 11 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 15 client.4422.objecter _finish_op 11 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 11 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1a802640 20 librados: complete_aio_write 0x55fa749d5f20 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (59 ms) 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74a90790 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.196 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 12 osd.7 2024-03-25T02:02:05.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 12 2024-03-25T02:02:05.209 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 15 client.4422.objecter _send_op 12 to 9.1 on osd.7 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:12 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a90c40 con 0x55fa749d63d0 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74a90fe0 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 13 osd.7 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 13 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 15 client.4422.objecter _send_op 13 to 9.1 on osd.7 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:13 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a92e20 con 0x55fa749d63d0 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:05.210 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.193+0000 7fcc1cc37780 5 client.4422.objecter 2 in flight 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 12 ondisk uv 4 in 9.1 attempt 0 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 5 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 12 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 15 client.4422.objecter _finish_op 12 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 12 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 13 ondisk uv 0 in 9.1 attempt 0 2024-03-25T02:02:05.211 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 10 client.4422.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 13 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 15 client.4422.objecter _finish_op 13 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (15 ms) 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 13 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74a90930 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.207+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.212 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 14 osd.7 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 14 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 15 client.4422.objecter _send_op 14 to 9.1 on osd.7 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:14 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a90de0 con 0x55fa749d63d0 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74a92e20 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:05.213 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-03-25T02:02:05.219 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.219 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.219 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 15 osd.7 2024-03-25T02:02:05.219 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 15 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 15 client.4422.objecter _send_op 15 to 9.1 on osd.7 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:15 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a91320 con 0x55fa749d63d0 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.208+0000 7fcc1cc37780 5 client.4422.objecter 2 in flight 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 14 ondisk uv 4 in 9.1 attempt 0 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 5 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 14 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 15 client.4422.objecter _finish_op 14 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 14 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.220 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-03-25T02:02:05.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 15 ondisk uv 0 in 9.1 attempt 0 2024-03-25T02:02:05.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 10 client.4422.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-25T02:02:05.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 15 2024-03-25T02:02:05.221 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 15 client.4422.objecter _finish_op 15 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 15 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.217+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (10 ms) 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74ab2e30 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:05.230 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 16 osd.7 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 16 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 15 client.4422.objecter _send_op 16 to 9.1 on osd.7 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:16 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a91320 con 0x55fa749d63d0 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74ad3400 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.231 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 17 osd.7 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 17 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 15 client.4422.objecter _send_op 17 to 9.1 on osd.7 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:17 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39) -- 0x55fa74ad38b0 con 0x55fa749d63d0 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.218+0000 7fcc1cc37780 5 client.4422.objecter 2 in flight 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 16 ondisk uv 4 in 9.1 attempt 0 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 5 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 16 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 15 client.4422.objecter _finish_op 16 2024-03-25T02:02:05.232 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 5 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 16 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 17 ondisk uv 0 in 9.1 attempt 0 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 10 client.4422.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 17 2024-03-25T02:02:05.233 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 15 client.4422.objecter _finish_op 17 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 17 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (11 ms) 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 20 librados: queue_aio_write 0x55fa7487ec30 completion 0x55fa74a91210 write_seq 4 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74a90930 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.234 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.276 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 18 osd.7 2024-03-25T02:02:05.276 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.276 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 18 2024-03-25T02:02:05.276 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 15 client.4422.objecter _send_op 18 to 9.1 on osd.7 2024-03-25T02:02:05.276 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:18 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a91320 con 0x55fa749d63d0 2024-03-25T02:02:05.276 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.228+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.274+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 18 ==== osd_op_reply(18 exist [writefull 0~5] v39'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.274+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(18 exist [writefull 0~5] v39'5 uv5 ondisk = 0) 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.274+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.274+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 18 ondisk uv 5 in 9.1 attempt 0 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 18 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1a802640 15 client.4422.objecter _finish_op 18 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 18 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1a802640 20 librados: complete_aio_write 0x55fa74a91210 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (46 ms) 2024-03-25T02:02:05.277 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 20 librados: queue_aio_write 0x55fa7487ec30 completion 0x55fa74a90820 write_seq 5 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74a90930 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 19 osd.7 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.278 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 19 2024-03-25T02:02:05.301 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 15 client.4422.objecter _send_op 19 to 9.1 on osd.7 2024-03-25T02:02:05.301 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:19 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a93e30 con 0x55fa749d63d0 2024-03-25T02:02:05.301 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.275+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 19 ==== osd_op_reply(19 exist [writefull 0~5] v39'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(19 exist [writefull 0~5] v39'6 uv6 ondisk = 0) 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 19 ondisk uv 6 in 9.1 attempt 0 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 19 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 15 client.4422.objecter _finish_op 19 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 19 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.299+0000 7fcc1a802640 20 librados: complete_aio_write 0x55fa74a90820 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (25 ms) 2024-03-25T02:02:05.302 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 20 librados: queue_aio_write 0x55fa7487ec30 completion 0x55fa74a91400 write_seq 6 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 10 client.4422.objecter _op_submit op 0x55fa74ab4490 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 20 client.4422.objecter _calc_target epoch 39 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 20 client.4422.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 10 client.4422.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [7,4] primary 7 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 20 client.4422.objecter _get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 10 client.4422.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 20 osd.7 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 20 client.4422.objecter get_session s=0x55fa749cc890 osd=7 3 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 15 client.4422.objecter _session_op_assign 7 20 2024-03-25T02:02:05.303 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 15 client.4422.objecter _send_op 20 to 9.1 on osd.7 2024-03-25T02:02:05.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- osd_op(unknown.0.0:20 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) -- 0x55fa74a93e30 con 0x55fa749d63d0 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.300+0000 7fcc1cc37780 5 client.4422.objecter 1 in flight 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 <== osd.7 v2:172.21.15.153:6826/2580901001 20 ==== osd_op_reply(20 exist [writefull 0~5] v39'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fcc0c009d50 con 0x55fa749d63d0 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 10 client.4422.objecter ms_dispatch 0x55fa748b5f70 osd_op_reply(20 exist [writefull 0~5] v39'7 uv7 ondisk = 0) 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 10 client.4422.objecter in handle_osd_op_reply 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 7 client.4422.objecter handle_osd_op_reply 20 ondisk uv 7 in 9.1 attempt 0 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 10 client.4422.objecter op 0 rval 0 len 0 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 15 client.4422.objecter handle_osd_op_reply completed tid 20 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 15 client.4422.objecter _finish_op 20 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 4 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 15 client.4422.objecter _session_op_remove 7 20 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 5 client.4422.objecter 0 in flight 2024-03-25T02:02:05.326 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 20 librados: complete_aio_write 0x55fa74a91400 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (24 ms) 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 10 librados: watch_flush enter 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 10 librados: watch_flush exit 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 10 client.4422.objecter close_session for osd.7 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x55fa749d63d0 msgr2=0x55fa74a7ac90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x55fa749d63d0 0x55fa74a7ac90 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 20 client.4422.objecter put_session s=0x55fa749cc890 osd=7 1 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 10 client.4422.objecter successfully canceled tick 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fcbfc06bce0 msgr2=0x7fcbfc06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fcbfc06bce0 0x7fcbfc06e1a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcc10058000 tx=0x7fcc10050070 comp rx=0 tx=0).stop 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 10 monclient: shutdown 2024-03-25T02:02:05.327 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa7494b680 msgr2=0x55fa749e1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 --2- 172.21.15.83:0/2311987957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fa7494b680 0x55fa749e1e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc14006480 tx=0x7fcc14001250 comp rx=0 tx=0).stop 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1a802640 1 -- 172.21.15.83:0/2311987957 reap_dead start 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 shutdown_connections 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 >> 172.21.15.83:0/2311987957 conn(0x55fa74908520 msgr2=0x55fa749465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 shutdown_connections 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 -- 172.21.15.83:0/2311987957 wait complete. 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.324+0000 7fcc1cc37780 1 librados: shutdown 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] 12 tests from AsioRados (3981 ms total) 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [----------] Global test environment tear-down 2024-03-25T02:02:05.329 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (31489 ms total) 2024-03-25T02:02:05.330 INFO:tasks.workunit.client.0.smithi083.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-25T02:02:05.716 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 55.854% pgs not active 2024-03-25T02:02:05.716 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 591 creating+peering 2024-03-25T02:02:05.716 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 460 unknown 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 53 creating+activating 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 49 active+clean 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: io: 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: client: 178 B/s wr, 0 op/s rd, 0 op/s wr 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5932 ms) 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ OK ] LibRadosService.Status (20455 ms) 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] 4 tests from LibRadosService (31699 ms total) 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [----------] Global test environment tear-down 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31699 ms total) 2024-03-25T02:02:05.717 INFO:tasks.workunit.client.0.smithi083.stdout: api_service: [ PASSED ] 4 tests. 2024-03-25T02:02:06.285 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5440 ms) 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (573 ms) 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20591 ms) 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: attempt 0 of 20 2024-03-25T02:02:06.286 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5636 ms) 2024-03-25T02:02:06.287 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (32240 ms total) 2024-03-25T02:02:06.287 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: 2024-03-25T02:02:06.287 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-25T02:02:06.287 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (32240 ms total) 2024-03-25T02:02:06.287 INFO:tasks.workunit.client.0.smithi083.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (15605 ms) 2024-03-25T02:02:20.365 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (9740 ms) 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (3303 ms) 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (545 ms) 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (1728 ms) 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (1035 ms) 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (182 ms) 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (147 ms) 2024-03-25T02:02:20.366 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (90 ms) 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (212 ms) 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (199 ms) 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (116 ms) 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (579 ms) 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (366 ms) 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-25T02:02:20.367 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (381 ms) 2024-03-25T02:02:20.368 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-25T02:02:20.368 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (71 ms) 2024-03-25T02:02:20.368 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (34299 ms total) 2024-03-25T02:02:20.368 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: 2024-03-25T02:02:20.368 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-25T02:02:20.368 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (46333 ms total) 2024-03-25T02:02:20.368 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-25T02:02:20.633 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (457 ms) 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (493 ms) 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (45703 ms) 2024-03-25T02:02:20.634 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (46653 ms total) 2024-03-25T02:02:20.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: 2024-03-25T02:02:20.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-25T02:02:20.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (46653 ms total) 2024-03-25T02:02:20.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-25T02:02:20.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-25T02:02:20.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-25T02:02:20.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-25T02:02:20.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-25T02:02:20.665 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (46588 ms) 2024-03-25T02:02:20.665 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (46588 ms total) 2024-03-25T02:02:20.665 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: 2024-03-25T02:02:20.665 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-25T02:02:20.665 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (46588 ms total) 2024-03-25T02:02:20.665 INFO:tasks.workunit.client.0.smithi083.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1334 ms) 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (720 ms) 2024-03-25T02:02:22.579 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (44614 ms) 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:12.006448+0000 mon.b [INF] from='client.4610 172.21.15.83:0/2553746377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40966-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:15.495154+0000 mon.a [INF] from='client.4610 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40966-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:16.292855+0000 mon.c [INF] from='client.? 172.21.15.83:0/320209320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40944-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:18.245934+0000 mon.a [WRN] Health check update: 20 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:18.298133+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40944-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:18.441237+0000 mon.a [INF] from='client.? 172.21.15.83:0/3817628094' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-41322-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:18.488267+0000 mon.c [INF] from='client.? 172.21.15.83:0/3545415271' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-41049-7"}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:19.602203+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi083-41049-7"}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.009572+0000 mon.a [INF] from='client.4610 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40966-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.430424+0000 mon.c [INF] from='client.? 172.21.15.83:0/1744437041' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-40860-10"}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.440768+0000 mon.c [INF] from='client.? 172.21.15.83:0/3545415271' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-41049-7"}]: dispatch 2024-03-25T02:02:22.580 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.688316+0000 mon.c [INF] from='client.? 172.21.15.83:0/1375237363' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-41125-7"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.747910+0000 mon.c [INF] from='client.? 172.21.15.83:0/629755693' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-40741-22"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.840078+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi083-40860-10"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.845917+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi083-41049-7"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.850923+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi083-41125-7"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.856717+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi083-40741-22"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.862122+0000 mon.b [INF] from='client.? 172.21.15.83:0/1585694159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-40790-10"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.872591+0000 mon.c [INF] from='client.? 172.21.15.83:0/3394449917' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.958053+0000 mon.c [INF] from='client.? 172.21.15.83:0/1744437041' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-40860-10"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.960613+0000 mon.a [INF] from='client.? 172.21.15.83:0/2516429058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi083-40719-16"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.967518+0000 mon.c [INF] from='client.? 172.21.15.83:0/629755693' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-40741-22"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.980640+0000 mon.c [INF] from='client.? 172.21.15.83:0/3545415271' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-41049-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:20.982933+0000 mon.c [INF] from='client.? 172.21.15.83:0/1375237363' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-41125-7"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.081605+0000 mon.a [INF] from='client.? 172.21.15.83:0/2516429058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi083-40719-16"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.180954+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi083-40790-10"}]: dispatch 2024-03-25T02:02:22.581 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.591881+0000 client.admin [INF] onexx 2024-03-25T02:02:22.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.614085+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-25T02:02:22.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.634946+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi083-40860-10"}]: dispatch 2024-03-25T02:02:22.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.639281+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi083-40741-22"}]: dispatch 2024-03-25T02:02:22.582 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.642985+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi083-41049-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-25T02:02:26.317 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.648535+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi083-41125-7"}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.677002+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40944-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:21.679099+0000 mon.a [INF] from='client.? 172.21.15.83:0/3817628094' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-41322-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:22.065578+0000 mon.a [INF] from='client.? 172.21.15.83:0/2516429058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi083-40719-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:22.110502+0000 mon.b [INF] from='client.? 172.21.15.83:0/1585694159' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-40790-10"}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:22.188022+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi083-40790-10"}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:22.230818+0000 mon.b [INF] from='client.? 172.21.15.83:0/1585694159' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-40790-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:22.247857+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi083-40790-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:23.232083+0000 mon.b [INF] from='client.? 172.21.15.83:0/1585694159' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-40790-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-40790-10"}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:23.683857+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-40790-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-40790-10"}]: dispatch 2024-03-25T02:02:26.318 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:23.708347+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi083-40741-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-25T02:02:26.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:23.710964+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi083-40860-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-25T02:02:26.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:23.714174+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi083-41125-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-25T02:02:26.319 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.034359+0000 mon.c [INF] from='client.? 172.21.15.83:0/629755693' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-40741-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-40741-22"}]: dispatch 2024-03-25T02:02:26.381 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.043052+0000 mon.c [INF] from='client.? 172.21.15.83:0/1744437041' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-40860-10", "pool_type":"era pool: Running main() from gmock_main.cc 2024-03-25T02:02:26.381 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-25T02:02:26.381 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [----------] Global test environment set-up. 2024-03-25T02:02:26.381 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-25T02:02:26.381 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ OK ] NeoRadosPools.PoolList (4171 ms) 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4419 ms) 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4105 ms) 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (29639 ms) 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (6688 ms) 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (3133 ms) 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [----------] 6 tests from NeoRadosPools (52156 ms total) 2024-03-25T02:02:26.382 INFO:tasks.workunit.client.0.smithi083.stdout: pool: 2024-03-25T02:02:26.383 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [----------] Global test environment tear-down 2024-03-25T02:02:26.383 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [==========] 6 tests from 1 test suite ran. (52156 ms total) 2024-03-25T02:02:26.383 INFO:tasks.workunit.client.0.smithi083.stdout: pool: [ PASSED ] 6 tests. 2024-03-25T02:02:26.663 INFO:tasks.workunit.client.0.smithi083.stdout:sure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-40860-10"}]: dispatch 2024-03-25T02:02:26.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.047232+0000 mon.c [INF] from='client.? 172.21.15.83:0/1375237363' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-41125-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-41125-7"}]: dispatch 2024-03-25T02:02:26.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.053586+0000 mon.c [INF] from='client.? 172.21.15.83:0/320209320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40944-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:26.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.136062+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-25T02:02:26.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.321519+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-40741-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-40741-22"}]: dispatch 2024-03-25T02:02:26.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.330857+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-40860-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-40860-10"}]: dispatch 2024-03-25T02:02:26.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.341039+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-41125-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-41125-7"}]: dispatch 2024-03-25T02:02:26.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.351321+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40944-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:26.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.873692+0000 mon.a [INF] from='client.? 172.21.15.83:0/2516429058' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi083-40719-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi083-40719-16"}]': finished 2024-03-25T02:02:26.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.875463+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi083-41049-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi083-41049-7"}]': finished 2024-03-25T02:02:26.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:24.877536+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40944-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-25T02:02:26.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:25.183882+0000 mon.c [INF] from='client.? 172.21.15.83:0/320209320' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-40944-4", "tierpool": "test-rados-api-smithi083-40944-4-cache"}]: dispatch 2024-03-25T02:02:26.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:25.249189+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-40944-4", "tierpool": "test-rados-api-smithi083-40944-4-cache"}]: dispatch 2024-03-25T02:02:26.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:25.354777+0000 mon.a [INF] from='client.? 172.21.15.83:0/1712222142' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40966-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-25T02:02:28.933 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:11.726 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:28.939 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:11.732 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:28.943 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:11.780 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:28.949 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:11.786 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:29.029 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:11.865 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:29.033 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:11.870 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:29.129 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:11.966 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:29.162 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:11.956 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:29.215 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:12.052 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:29.764 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.022077+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi083-40790-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi083-40790-10"}]': finished 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.028144+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi083-40741-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi083-40741-22"}]': finished 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.034007+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi083-40860-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi083-40860-10"}]': finished 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.037932+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi083-41125-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi083-41125-7"}]': finished 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.041226+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi083-40944-4", "tierpool": "test-rados-api-smithi083-40944-4-cache"}]': finished 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.044169+0000 mon.a [INF] from='client.? 172.21.15.83:0/1712222142' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi083-40966-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.287909+0000 mon.c [INF] from='client.? 172.21.15.83:0/3394449917' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.334909+0000 client.admin [INF] threexx 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.335501+0000 mon.c [INF] from='client.? 172.21.15.83:0/320209320' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-40944-4", "overlaypool": "test-rados-api-smithi083-40944-4-cache"}]: dispatch 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.349292+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: got: 2024-03-25T02:02:26.364632+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi083-40944-4", "overlaypool": "test-rados-api-smithi083-40944-4-cache"}]: dispatch 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (9135 ms) 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (55803 ms total) 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: 2024-03-25T02:02:29.765 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-25T02:02:29.766 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (55803 ms total) 2024-03-25T02:02:29.766 INFO:tasks.workunit.client.0.smithi083.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-25T02:02:32.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-25T02:02:32.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] Global test environment set-up. 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-03-25T02:01:34.073+0000 7fe21a23e780 -1 failed for service _ceph-mon._tcp 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-03-25T02:01:34.073+0000 7fe21a23e780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (250 ms) 2024-03-25T02:02:32.165 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2017 ms) 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2267 ms total) 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940e70d810 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940e968050 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940e968fb0 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940f315ae0 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940e9f6ce0 2024-03-25T02:02:32.166 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea99980 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9a380 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9ad80 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9b820 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9be80 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9c8b0 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9b510 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa5c20 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa6710 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9c690 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa76a0 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa6470 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa8580 2024-03-25T02:02:32.167 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa73d0 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa94d0 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa8330 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaaa3d0 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa91c0 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaaaf60 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaab680 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaac130 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaaa110 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea99100 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9c7a0 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa6580 2024-03-25T02:02:32.168 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9b620 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9c520 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea987c0 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea99210 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa8460 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea99320 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea99430 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9b930 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9ba80 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9bbd0 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9bce0 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaab110 2024-03-25T02:02:32.169 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaab220 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaab370 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaab480 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9b340 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa61e0 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaabc60 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaabd70 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaabe80 2024-03-25T02:02:32.170 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaabf90 2024-03-25T02:02:49.444 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:32.280 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.470 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:32.307 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.472 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:32.265 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.477 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:32.270 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.484 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:32.322 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.491 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:32.328 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.720 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:32.557 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.733 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:32.527 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:49.782 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:32.619 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:02:53.359 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:02:53.220+0000 ea52640 -1 osd.0 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: starte api_pool: Running main() from gmock_main.cc 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] Global test environment set-up. 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolList (8974 ms) 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3757 ms) 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (10356 ms) 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-25T02:02:54.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (26315 ms) 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4227 ms) 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5687 ms) 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5947 ms) 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6229 ms) 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9077 ms) 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] 9 tests from LibRadosPools (80569 ms total) 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: 2024-03-25T02:02:54.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [----------] Global test environment tear-down 2024-03-25T02:02:54.431 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (80569 ms total) 2024-03-25T02:02:54.431 INFO:tasks.workunit.client.0.smithi083.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-25T02:02:54.511 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:02:54.343+0000 ea52640 -1 osd.3 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 65.13 65.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-03-25T02:02:54.560 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:02:54.439+0000 ea52640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:02:55.583 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:02:55.510+0000 ea52640 -1 osd.0 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:02:55.623 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:02:55.506+0000 ea52640 -1 osd.3 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 65.13 65.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-03-25T02:02:56.529 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:02:56.521+0000 ea52640 -1 osd.3 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 65.13 65.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-03-25T02:02:56.691 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:02:56.673+0000 ea52640 -1 osd.0 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:02:57.938 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:02:57.892+0000 ea52640 -1 osd.0 70 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:02:57.967 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:02:57.903+0000 ea52640 -1 osd.3 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4762.0:2 65.13 65.be9754b3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-03-25T02:02:59.394 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:02:59.322+0000 ea52640 -1 osd.0 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:00.360 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:00.357+0000 ea52640 -1 osd.0 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:01.401 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:01.399+0000 ea52640 -1 osd.0 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:01.570 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:01.464+0000 ea52640 -1 osd.3 76 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4762.0:2 65.13 65:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-03-25T02:03:02.625 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:02.610+0000 ea52640 -1 osd.0 76 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:02.993 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:02.984+0000 ea52640 -1 osd.3 77 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4762.0:2 65.13 65:cd2ae97d:::foo1:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) 2024-03-25T02:03:03.712 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:03.706+0000 ea52640 -1 osd.0 78 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:04.017 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:04.010+0000 ea52640 -1 osd.3 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:04.835 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:04.739+0000 ea52640 -1 osd.0 79 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:05.106 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:05.022+0000 ea52640 -1 osd.3 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:05.838 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:05.798+0000 ea52640 -1 osd.0 80 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:07.020 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:07.016+0000 ea52640 -1 osd.0 81 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:08.134 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:08.029+0000 ea52640 -1 osd.0 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:09.062 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:09.053+0000 ea52640 -1 osd.0 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:09.834 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:52.671 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:09.918 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:52.755 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:09.929 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:52.720 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:09.937 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:52.731 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:09.998 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:52.836 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:10.004 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:52.842 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:10.080 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:10.073+0000 ea52640 -1 osd.0 83 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:10.082 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:03:52.876 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:10.102 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:03:52.940 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:10.149 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:03:52.987 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:11.376 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:11.305+0000 ea52640 -1 osd.0 85 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:12.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-25T02:03:12.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-25T02:03:12.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-25T02:03:12.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-25T02:03:12.099 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify_test_cb 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (22568 ms) 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94260446622224 err -107 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (4543 ms) 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94260446641024 err -107 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1531 ms) 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 167503724544 cookie 94260446654272 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (1965 ms) 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-25T02:03:12.100 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 167503724545 cookie 94260446692720 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (2060 ms) 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 171798691842 cookie 94260446692720 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (828 ms) 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 171798691840 cookie 94260446673712 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 171798691840 cookie 94260446694160 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (408 ms) 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 171798691840 cookie 94260446673712 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 171798691841 cookie 94260446673712 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3255 ms) 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-25T02:03:12.101 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94260446673712 err -107 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_cb from 4445 notify_id 193273528324 cookie 94260446673712 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (28392 ms) 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94260446694112 err -107 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (6663 ms) 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (72213 ms total) 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: watch_notify_test_cb 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (14563 ms) 2024-03-25T02:03:12.102 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (14563 ms total) 2024-03-25T02:03:12.103 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: 2024-03-25T02:03:12.103 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-25T02:03:12.103 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (98134 ms total) 2024-03-25T02:03:12.103 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-25T02:03:12.509 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:12.345+0000 ea52640 -1 osd.0 86 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:12.859 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:12.740+0000 ea52640 -1 osd.3 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:13.528 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:13.521+0000 ea52640 -1 osd.0 87 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:14.839 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:14.824+0000 ea52640 -1 osd.0 88 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:15.118 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:15.094+0000 ea52640 -1 osd.3 86 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:15.941 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:15.907+0000 ea52640 -1 osd.0 90 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:16.232 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:16.219+0000 ea52640 -1 osd.3 88 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:17.085 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:17.077+0000 ea52640 -1 osd.0 91 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:17.517 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:17.506+0000 ea52640 -1 osd.3 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:18.314 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:18.233+0000 ea52640 -1 osd.0 92 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:19.473 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:19.435+0000 ea52640 -1 osd.0 93 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:19.477 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:19.453+0000 12a5a640 -1 osd.7 94 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:20.499 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:20.494+0000 12a5a640 -1 osd.7 94 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:20.955 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:20.920+0000 ea52640 -1 osd.0 94 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:21.498 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:21.490+0000 12a5a640 -1 osd.7 95 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:22.159 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:22.066+0000 ea52640 -1 osd.0 95 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:22.741 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:22.659+0000 12a5a640 -1 osd.7 96 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:23.430 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:23.229+0000 ea52640 -1 osd.0 96 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4652.0:2 61.15 61.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:23.814 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:23.718+0000 12a5a640 -1 osd.7 97 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:24.668 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:24.549+0000 ea52640 -1 osd.3 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:24.676 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:24.664+0000 ea52640 -1 osd.0 96 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4652.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:24.897 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:24.875+0000 12a5a640 -1 osd.7 98 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:26.112 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:26.103+0000 12a5a640 -1 osd.7 100 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:26.137 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:26.089+0000 ea52640 -1 osd.0 97 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4652.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:26.635 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:26.630+0000 ea52640 -1 osd.3 98 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:27.139 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:27.125+0000 ea52640 -1 osd.0 98 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4652.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:27.513 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:27.500+0000 12a5a640 -1 osd.7 101 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:28.259 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:28.150+0000 ea52640 -1 osd.0 99 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4652.0:2 61.15 61:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43) 2024-03-25T02:03:28.720 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:28.655+0000 12a5a640 -1 osd.7 102 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:29.454 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:29.405+0000 ea52640 -1 osd.3 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:29.778 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:12.616 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.781 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:12.618 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.839 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:12.634 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.844 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:12.638 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.853 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:12.691 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.857 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:12.695 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.920 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:04:12.758 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.952 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:04:12.746 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:29.972 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:29.922+0000 12a5a640 -1 osd.7 103 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:30.009 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:04:12.847 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:03:31.111 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:31.025+0000 12a5a640 -1 osd.7 105 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:31.349 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: seed 40741 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (28517 ms) 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-25T02:03:31.350 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (277 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (84 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1025 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (60 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (59 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (1723 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (117 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (113 ms) 2024-03-25T02:03:31.351 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (182 ms) 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (1722 ms) 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (474 ms) 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (402 ms) 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (266 ms) 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (527 ms) 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (236 ms) 2024-03-25T02:03:31.352 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (222 ms) 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (99 ms) 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (190 ms) 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (36295 ms total) 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (37408 ms) 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (11639 ms) 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-25T02:03:31.353 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (2382 ms) 2024-03-25T02:03:31.354 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-25T02:03:31.354 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (6413 ms) 2024-03-25T02:03:31.354 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-25T02:03:32.192 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:32.142+0000 12a5a640 -1 osd.7 106 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:33.195 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:33.193+0000 12a5a640 -1 osd.7 106 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:34.287 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:34.284+0000 12a5a640 -1 osd.7 107 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:34.943 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:34.923+0000 ea52640 -1 osd.0 105 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70.c72b327f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:35.267 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:35.264+0000 12a5a640 -1 osd.7 108 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:35.999 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:35.993+0000 ea52640 -1 osd.0 106 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70.c72b327f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:37.156 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:37.151+0000 ea52640 -1 osd.0 107 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70.c72b327f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:37.517 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:03:37.427+0000 ea52640 -1 osd.3 106 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4765.0:1 73.1s0 73.1b575219 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e47) 2024-03-25T02:03:37.611 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:37.590+0000 12a5a640 -1 osd.7 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:40.073 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:40.028+0000 ea52640 -1 osd.0 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70.c72b327f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:40.553 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:40.421+0000 12a5a640 -1 osd.7 114 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:41.398 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:41.393+0000 12a5a640 -1 osd.7 114 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:41.957 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:41.856+0000 ea52640 -1 osd.0 112 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70:fe4cd4e3:test-rados-api-smithi083-41049-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:42.389 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:42.386+0000 12a5a640 -1 osd.7 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:42.834 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:42.829+0000 ea52640 -1 osd.0 113 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70:fe4cd4e3:test-rados-api-smithi083-41049-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:43.659 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:43.608+0000 12a5a640 -1 osd.7 116 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4631.0:4 68.6 68:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 api_stat_pp: Running main() from gmock_main.cc 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: seed 41125 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (25344 ms) 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (2781 ms) 2024-03-25T02:03:44.059 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1252 ms) 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1771 ms) 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (1393 ms) 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (32541 ms total) 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (41245 ms) 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (4 ms) 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-25T02:03:44.060 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (11708 ms) 2024-03-25T02:03:44.061 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-25T02:03:44.061 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (21937 ms) 2024-03-25T02:03:44.061 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (74894 ms total) 2024-03-25T02:03:44.061 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: 2024-03-25T02:03:44.061 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-25T02:03:44.061 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (130135 ms total) 2024-03-25T02:03:44.061 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-25T02:03:44.098 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:44.074+0000 ea52640 -1 osd.0 114 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70:fe4cd4e3:test-rados-api-smithi083-41049-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:44.620 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:03:44.614+0000 12a5a640 -1 osd.7 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4631.0:5 68.6 68:602f83fe:::foo:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61) 2024-03-25T02:03:45.664 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:45.548+0000 ea52640 -1 osd.0 115 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70:fe4cd4e3:test-rados-api-smithi083-41049-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:46.718 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:46.604+0000 ea52640 -1 osd.0 117 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4661.0:1 70.7s0 70:fe4cd4e3:test-rados-api-smithi083-41049-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) 2024-03-25T02:03:47.843 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:47.838+0000 ea52640 -1 osd.0 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4757.0:2 92.0 92:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-03-25T02:03:48.943 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:48.831+0000 ea52640 -1 osd.0 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4757.0:2 92.0 92:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-03-25T02:03:50.202 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:50.104+0000 ea52640 -1 osd.0 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4757.0:2 92.0 92:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-03-25T02:03:52.350 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:03:52.288+0000 ea52640 -1 osd.0 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4757.0:2 92.0 92:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79) 2024-03-25T02:03:56.172 INFO:tasks.workunit.client.0.smithi083.stdout:d 0x55940ea9a490 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea988d0 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea989e0 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea98af0 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9bf90 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9c0c0 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9c270 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa86b0 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940eaa8d10 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 0x55940ea9b230 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: started 61 aios 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940e70d810 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940e968050 2024-03-25T02:03:56.173 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940e968fb0 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940f315ae0 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940e9f6ce0 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea99980 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9a380 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9ad80 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9b820 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9be80 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9c8b0 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9b510 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa5c20 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa6710 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9c690 2024-03-25T02:03:56.174 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa76a0 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa6470 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa8580 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa73d0 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa94d0 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa8330 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaaa3d0 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa91c0 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaaaf60 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaab680 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaac130 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaaa110 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea99100 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9c7a0 2024-03-25T02:03:56.175 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa6580 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9b620 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9c520 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea987c0 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea99210 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa8460 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea99320 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea99430 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9b930 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9ba80 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9bbd0 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9bce0 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaab110 2024-03-25T02:03:56.176 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaab220 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaab370 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaab480 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9b340 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa61e0 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaabc60 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaabd70 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaabe80 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaabf90 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9a490 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea988d0 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea989e0 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea98af0 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9bf90 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9c0c0 2024-03-25T02:03:56.177 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9c270 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa86b0 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940eaa8d10 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: waiting 0x55940ea9b230 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: done. 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (69548 ms) 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (69548 ms total) 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-25T02:03:56.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-25T02:04:02.881 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] Li cls: Running main() from gmock_main.cc 2024-03-25T02:04:02.881 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-25T02:04:02.881 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [----------] Global test environment set-up. 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [ OK ] NeoRadosCls.DNE (42174 ms) 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (106601 ms) 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [----------] 2 tests from NeoRadosCls (148775 ms total) 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [----------] Global test environment tear-down 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [==========] 2 tests from 1 test suite ran. (148776 ms total) 2024-03-25T02:04:02.882 INFO:tasks.workunit.client.0.smithi083.stdout: cls: [ PASSED ] 2 tests. 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: Running main() from gmock_main.cc 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] Global test environment set-up. 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.Stat (21088 ms) 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (332 ms) 2024-03-25T02:04:03.952 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.StatNS (9916 ms) 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (13 ms) 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (83 ms) 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 5 tests from LibRadosStat (31432 ms total) 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (85286 ms) 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (7702 ms) 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-25T02:04:03.953 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (33 ms) 2024-03-25T02:04:03.954 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-25T02:04:03.954 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2533 ms) 2024-03-25T02:04:03.954 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (95554 ms total) 2024-03-25T02:04:03.954 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: 2024-03-25T02:04:03.954 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [----------] Global test environment tear-down 2024-03-25T02:04:03.954 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (150066 ms total) 2024-03-25T02:04:03.954 INFO:tasks.workunit.client.0.smithi083.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-25T02:04:17.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-25T02:04:17.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-25T02:04:17.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-25T02:04:17.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-25T02:04:17.296 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (25647 ms) 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (39030 ms) 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (64677 ms total) 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (9871 ms) 2024-03-25T02:04:17.297 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (36247 ms) 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (6150 ms) 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (6446 ms) 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362587600 notify_id 558345748483 notifier_gid 4751 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (18121 ms) 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-25T02:04:17.298 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362574896 notify_id 597000454148 notifier_gid 4751 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (4122 ms) 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362565328 notify_id 597000454144 notifier_gid 4751 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (1128 ms) 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362050880 notify_id 597000454145 notifier_gid 4751 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (243 ms) 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362050752 notify_id 601295421445 notifier_gid 4751 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (1425 ms) 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362050752 notify_id 601295421446 notifier_gid 4751 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (306 ms) 2024-03-25T02:04:17.299 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-25T02:04:17.300 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: trying... 2024-03-25T02:04:17.300 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362050752 notify_id 601295421445 notifier_gid 4751 2024-03-25T02:04:17.300 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: timed out 2024-03-25T02:04:18.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-25T02:04:18.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] Global test environment set-up. 2024-03-25T02:04:18.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-25T02:04:18.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-25T02:04:18.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjects (31751 ms) 2024-03-25T02:04:18.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-25T02:04:18.084 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (3822 ms) 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo2,foo3 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo2 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo3 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo4,foo5 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo4 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo5 2024-03-25T02:04:18.085 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo6,foo7 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo7 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo6 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo4 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo5 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo7 2024-03-25T02:04:18.086 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo6 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo1 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo1 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo2 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo3 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (2395 ms) 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 1 0 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 10 0 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 13 0 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 7 0 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 14 0 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 0 0 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 15 0 2024-03-25T02:04:18.087 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 11 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 5 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 8 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 6 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 3 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 4 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 12 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 9 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2 0 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (96394 ms) 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: x cursor=MIN 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-03-25T02:04:18.088 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-03-25T02:04:18.089 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:566253c9:::13:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-03-25T02:04:18.090 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-03-25T02:04:18.322 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:18.313+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:19.354 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:19.345+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:19.894 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-03-25T02:04:19.894 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-03-25T02:04:19.894 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-03-25T02:04:19.894 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-03-25T02:04:19.894 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-03-25T02:04:19.894 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:863748b0:::15:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-03-25T02:04:19.895 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:e960b815:::9:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-03-25T02:04:19.896 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:566253c9:::13:head 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=13 expected=13 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=10 expected=10 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-03-25T02:04:19.897 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=1 expected=1 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=4 expected=4 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=3 expected=3 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=6 expected=6 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-03-25T02:04:19.898 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-03-25T02:04:19.899 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-03-25T02:04:19.899 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=2 expected=2 2024-03-25T02:04:19.899 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:e960b815:::9:head 2024-03-25T02:04:19.899 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-03-25T02:04:19.899 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-03-25T02:04:19.899 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=9 expected=9 2024-03-25T02:04:19.899 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-03-25T02:04:20.346 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:20.341+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:21.349 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:21.345+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:22.339 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:22.334+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:23.373 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:23.366+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:24.381 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:24.376+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:25.343 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:25.340+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:26.344 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:26.334+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:27.375 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:27.373+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:28.056 INFO:tasks.ceph.mon.b.smithi153.stderr:2024-03-25T02:04:28.040+0000 ffa4640 -1 mon.b@1(peon).paxos(paxos updating c 1..417) lease_expire from mon.0 v2:172.21.15.83:3300/0 is 0.632179856s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-03-25T02:04:28.397 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:28.391+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:28.542 INFO:tasks.workunit.client.0.smithi083.stdout: a list_parallel: process_1_[41488]: starting. 2024-03-25T02:04:28.542 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[41488]: creating pool ceph_test_rados_list_parallel.smithi083-41448 2024-03-25T02:04:28.542 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[41488]: created object 0... 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[41488]: created object 25... 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[41488]: created object 49... 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[41488]: finishing. 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_1_[41488]: shutting down. 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[41536]: starting. 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[41536]: listing objects. 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[41536]: listed object 0... 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[41536]: listed object 25... 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[41536]: saw 50 objects 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_2_[41536]: shutting down. 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[42907]: starting. 2024-03-25T02:04:28.543 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[42907]: creating pool ceph_test_rados_list_parallel.smithi083-41448 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[42907]: created object 0... 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[42907]: created object 25... 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[42907]: created object 49... 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[42907]: finishing. 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_3_[42907]: shutting down. 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[42908]: starting. 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[42908]: listing objects. 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[42908]: listed object 0... 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[42908]: listed object 25... 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[42908]: saw 40 objects 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_4_[42908]: shutting down. 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.544 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[42909]: starting. 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[42909]: removed 25 objects... 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[42909]: removed half of the objects 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[42909]: removed 50 objects... 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[42909]: removed 50 objects 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_5_[42909]: shutting down. 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[52417]: starting. 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[52417]: creating pool ceph_test_rados_list_parallel.smithi083-41448 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[52417]: created object 0... 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[52417]: created object 25... 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[52417]: created object 49... 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[52417]: finishing. 2024-03-25T02:04:28.545 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_6_[52417]: shutting down. 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[52418]: starting. 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[52418]: listing objects. 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[52418]: listed object 0... 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[52418]: listed object 25... 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[52418]: listed object 50... 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[52418]: saw 53 objects 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_7_[52418]: shutting down. 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[52419]: starting. 2024-03-25T02:04:28.546 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[52419]: added 25 objects... 2024-03-25T02:04:28.547 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[52419]: added half of the objects 2024-03-25T02:04:28.547 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[52419]: added 50 objects... 2024-03-25T02:04:28.547 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[52419]: added 50 objects 2024-03-25T02:04:28.547 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_8_[52419]: shutting down. 2024-03-25T02:04:28.547 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:28.547 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:29.413 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:29.409+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:30.388 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:30.385+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:31.446 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:31.443+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:32.400 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:04:32.396+0000 ea52640 -1 osd.0 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4796.0:2 103.0 103.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e132) 2024-03-25T02:04:32.483 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:32.476+0000 ea52640 -1 osd.3 140 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:33.705 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:33.701+0000 ea52640 -1 osd.3 141 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:34.511 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:04:34.352+0000 ea52640 -1 osd.0 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4796.0:2 103.0 103.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e132) 2024-03-25T02:04:34.733 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:34.713+0000 ea52640 -1 osd.3 141 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:35.725 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:35.722+0000 ea52640 -1 osd.3 141 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:36.736 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:36.732+0000 ea52640 -1 osd.3 141 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:37.781 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:37.778+0000 ea52640 -1 osd.3 141 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:37.787 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ********************** open_pools_parallel: process_1_[41585]: starting. 2024-03-25T02:04:37.787 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[41585]: creating pool ceph_test_rados_open_pools_parallel.smithi083-41520 2024-03-25T02:04:37.787 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[41585]: created object 0... 2024-03-25T02:04:37.787 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[41585]: created object 25... 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[41585]: created object 49... 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[41585]: finishing. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_1_[41585]: shutting down. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[41605]: starting. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[41605]: rados_pool_create. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[41605]: rados_ioctx_create. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_2_[41605]: shutting down. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[43199]: starting. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[43199]: creating pool ceph_test_rados_open_pools_parallel.smithi083-41520 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[43199]: created object 0... 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[43199]: created object 25... 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[43199]: created object 49... 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[43199]: finishing. 2024-03-25T02:04:37.788 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_3_[43199]: shutting down. 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[43200]: starting. 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[43200]: rados_pool_create. 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[43200]: rados_ioctx_create. 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: process_4_[43200]: shutting down. 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******************************* 2024-03-25T02:04:37.789 INFO:tasks.workunit.client.0.smithi083.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-25T02:04:39.329 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:39.188+0000 ea52640 -1 osd.3 142 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4787.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:40.431 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:04:40.315+0000 ea52640 -1 osd.3 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4787.0:2 99.15 99:ad551702:::test:head [create,write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e104) 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: Running main() from gmock_main.cc 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] Global test environment set-up. 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (23476 ms) 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-25T02:04:42.909 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: no timeout :/ 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (4676 ms) 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (3068 ms) 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.Checksum (49 ms) 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-25T02:04:42.910 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (108 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (103 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (256 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (39 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.TruncTest (109 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (224 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (2753 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.RmXattr (1052 ms) 2024-03-25T02:04:42.911 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIo.XattrIter (393 ms) 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 14 tests from LibRadosIo (36306 ms total) 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (42476 ms) 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (25378 ms) 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (3230 ms) 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (10076 ms) 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-25T02:04:42.912 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (3418 ms) 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (9036 ms) 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3234 ms) 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (7208 ms) 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (2236 ms) 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (269 ms) 2024-03-25T02:04:42.913 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] 10 tests from LibRadosIoEC (106561 ms total) 2024-03-25T02:04:42.914 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: 2024-03-25T02:04:42.914 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [----------] Global test environment tear-down 2024-03-25T02:04:42.914 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [==========] 24 tests from 2 test suites ran. (189166 ms total) 2024-03-25T02:04:42.914 INFO:tasks.workunit.client.0.smithi083.stdout: api_io: [ PASSED ] 24 tests. 2024-03-25T02:04:43.674 INFO:tasks.workunit.client.0.smithi083.stdout:********* 2024-03-25T02:04:43.674 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[59506]: starting. 2024-03-25T02:04:43.674 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[59506]: creating pool ceph_test_rados_list_parallel.smithi083-41448 2024-03-25T02:04:43.674 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[59506]: created object 0... 2024-03-25T02:04:43.674 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[59506]: created object 25... 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[59506]: created object 49... 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[59506]: finishing. 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_9_[59506]: shutting down. 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: starting. 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: listing objects. 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: listed object 0... 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: listed object 25... 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: listed object 50... 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: listed object 75... 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: saw 99 objects 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_10_[59507]: shutting down. 2024-03-25T02:04:43.675 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[59510]: starting. 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[59510]: removed 25 objects... 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[59510]: removed half of the objects 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[59510]: removed 50 objects... 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[59510]: removed 50 objects 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_13_[59510]: shutting down. 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[59508]: starting. 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[59508]: added 25 objects... 2024-03-25T02:04:43.676 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[59508]: added half of the objects 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[59508]: added 50 objects... 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[59508]: added 50 objects 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_11_[59508]: shutting down. 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[59509]: starting. 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[59509]: added 25 objects... 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[59509]: added half of the objects 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[59509]: added 50 objects... 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[59509]: added 50 objects 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_12_[59509]: shutting down. 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.677 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[66991]: starting. 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[66991]: creating pool ceph_test_rados_list_parallel.smithi083-41448 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[66991]: created object 0... 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[66991]: created object 25... 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[66991]: created object 49... 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[66991]: finishing. 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_14_[66991]: shutting down. 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: starting. 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: listing objects. 2024-03-25T02:04:43.678 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: listed object 0... 2024-03-25T02:04:48.538 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: listed object 2 delete_pools_parallel: process_1_[41591]: starting. 2024-03-25T02:04:48.538 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[41591]: creating pool ceph_test_rados_delete_pools_parallel.smithi083-41539 2024-03-25T02:04:48.538 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[41591]: created object 0... 2024-03-25T02:04:48.538 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[41591]: created object 25... 2024-03-25T02:04:48.538 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[41591]: created object 49... 2024-03-25T02:04:48.538 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[41591]: finishing. 2024-03-25T02:04:48.538 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_1_[41591]: shutting down. 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_2_[41594]: starting. 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_2_[41594]: deleting pool ceph_test_rados_delete_pools_parallel.smithi083-41539 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_2_[41594]: shutting down. 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[43087]: starting. 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[43087]: creating pool ceph_test_rados_delete_pools_parallel.smithi083-41539 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[43087]: created object 0... 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[43087]: created object 25... 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[43087]: created object 49... 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[43087]: finishing. 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_3_[43087]: shutting down. 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-03-25T02:04:48.539 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[43089]: starting. 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[43089]: listing objects. 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[43089]: listed object 0... 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[43089]: listed object 25... 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[43089]: saw 50 objects 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_5_[43089]: shutting down. 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_4_[43088]: starting. 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_4_[43088]: deleting pool ceph_test_rados_delete_pools_parallel.smithi083-41539 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: process_4_[43088]: shutting down. 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******************************* 2024-03-25T02:04:48.540 INFO:tasks.workunit.client.0.smithi083.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-25T02:05:01.717 INFO:tasks.workunit.client.0.smithi083.stdout:5... 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: listed object 50... 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: listed object 75... 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: listed object 100... 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: listed object 125... 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: saw 150 objects 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_15_[66992]: shutting down. 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.718 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[66993]: starting. 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[66993]: added 25 objects... 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[66993]: added half of the objects 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[66993]: added 50 objects... 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[66993]: added 50 objects 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: process_16_[66993]: shutting down. 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.719 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******************************* 2024-03-25T02:05:01.720 INFO:tasks.workunit.client.0.smithi083.stdout: list_parallel: ******* SUCCESS ********** 2024-03-25T02:05:02.411 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: Running main() from gmock_main.cc 2024-03-25T02:05:02.411 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] Global test environment set-up. 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (16594 ms) 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockShared (7141 ms) 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (4664 ms) 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1712 ms) 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (1587 ms) 2024-03-25T02:05:02.412 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.Unlock (256 ms) 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (198 ms) 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (173 ms) 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLock (32326 ms total) 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (85167 ms) 2024-03-25T02:05:02.413 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (2064 ms) 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1236 ms) 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1838 ms) 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (16846 ms) 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (1648 ms) 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (889 ms) 2024-03-25T02:05:02.414 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-25T02:05:02.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (446 ms) 2024-03-25T02:05:02.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (110134 ms total) 2024-03-25T02:05:02.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: 2024-03-25T02:05:02.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [----------] Global test environment tear-down 2024-03-25T02:05:02.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (208637 ms total) 2024-03-25T02:05:02.415 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-25T02:05:06.156 INFO:tasks.workunit.client.0.smithi083.stdout: (7166 ms) 2024-03-25T02:05:06.156 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-25T02:05:06.156 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (6624 ms) 2024-03-25T02:05:06.156 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-25T02:05:06.156 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (2918 ms) 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (11580 ms) 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (21184 ms) 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (1974 ms) 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (408 ms) 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (442 ms) 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (837 ms) 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-25T02:05:06.157 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (223 ms) 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (447 ms) 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (167 ms) 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (251 ms) 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (112063 ms total) 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (212403 ms total) 2024-03-25T02:05:06.158 INFO:tasks.workunit.client.0.smithi083.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: seed 40860 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (21336 ms) 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (329 ms) 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (2306 ms) 2024-03-25T02:05:06.192 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (6291 ms) 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (96 ms) 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (1603 ms) 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (1892 ms) 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (805 ms) 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (34658 ms total) 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (99482 ms) 2024-03-25T02:05:06.193 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (7183 ms) 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1637 ms) 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1958 ms) 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (496 ms) 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (292 ms) 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (351 ms) 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-25T02:05:06.194 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (278 ms) 2024-03-25T02:05:06.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (111677 ms total) 2024-03-25T02:05:06.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: 2024-03-25T02:05:06.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-25T02:05:06.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (212384 ms total) 2024-03-25T02:05:06.195 INFO:tasks.workunit.client.0.smithi083.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: flushing 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushed 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3339 ms) 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: trying... 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362050752 notify_id 601295421446 notifier_gid 4751 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: timed out 2024-03-25T02:05:11.323 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushing 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushed 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3282 ms) 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: List watches 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362050752 notify_id 601295421447 notifier_gid 4751 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 done 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: watch_check 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: unwatch2 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushing 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: done 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3216 ms) 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-25T02:05:11.324 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: List watches 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: handle_notify cookie 94018362050752 notify_id 601295421442 notifier_gid 4751 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: notify2 done 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: watch_check 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: unwatch2 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: flushing 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: done 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3248 ms) 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (97145 ms total) 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (217369 ms total) 2024-03-25T02:05:11.325 INFO:tasks.workunit.client.0.smithi083.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout:bRadosMisc.Exec (67221 ms) 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (2611 ms) 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (575 ms) 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5638 ms) 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-25T02:05:15.663 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-25T02:05:15.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-25T02:05:15.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (69251 ms) 2024-03-25T02:05:15.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] 8 tests from LibRadosMisc (145296 ms total) 2024-03-25T02:05:15.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: 2024-03-25T02:05:15.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [----------] Global test environment tear-down 2024-03-25T02:05:15.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (221827 ms total) 2024-03-25T02:05:15.664 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-25T02:05:15.815 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:58.655 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:15.821 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:58.661 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:15.880 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:58.676 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:15.917 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:58.756 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:15.924 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:58.720 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:15.939 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:58.779 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:16.302 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:05:59.142 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:16.360 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:05:59.156 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:16.387 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:05:59.226 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.216 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:19.056 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.220 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:19.060 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.301 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:19.141 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.304 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:19.101 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.305 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:19.144 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.313 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:19.109 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.324 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:19.164 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.413 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:19.209 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:36.428 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:19.267 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.412 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:39.251 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.416 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:39.255 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.496 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:39.337 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.500 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:39.340 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.504 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:39.301 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.509 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:39.306 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.571 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:39.367 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.590 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:39.430 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:05:56.670 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:39.511 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:12.584 INFO:tasks.workunit.client.0.smithi083.stdout: io: Running main() from gmock_main.cc 2024-03-25T02:06:12.584 INFO:tasks.workunit.client.0.smithi083.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [----------] Global test environment set-up. 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.Limits (46423 ms) 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (63812 ms) 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.ReadOp (14972 ms) 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.SparseRead (6701 ms) 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.RoundTrip (45828 ms) 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-25T02:06:12.585 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (39039 ms) 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (13898 ms) 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (7255 ms) 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (7174 ms) 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.Trunc (5924 ms) 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.Remove (4760 ms) 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (6336 ms) 2024-03-25T02:06:12.586 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.RmXattr (8563 ms) 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ OK ] NeoRadosIo.GetXattrs (7698 ms) 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: [----------] 14 tests from NeoRadosIo (278383 ms total) 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: [----------] Global test environment tear-down 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: [==========] 14 tests from 1 test suite ran. (278384 ms total) 2024-03-25T02:06:12.587 INFO:tasks.workunit.client.0.smithi083.stdout: io: [ PASSED ] 14 tests. 2024-03-25T02:06:15.696 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: seed 41020 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (11819 ms) 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (27543 ms) 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (21273 ms) 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (7953 ms) 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-25T02:06:15.697 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (16072 ms) 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (84661 ms total) 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (12424 ms) 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (44568 ms) 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (71474 ms) 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (14946 ms) 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-25T02:06:15.698 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (10296 ms) 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2398-g9c429fb6/rpm/el9/BUILD/ceph-19.0.0-2398-g9c429fb6/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi083-41020-7 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19865 ms) 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (173573 ms total) 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-25T02:06:15.699 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (6 ms) 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5584 ms) 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5590 ms total) 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (6215 ms) 2024-03-25T02:06:15.700 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-25T02:06:16.410 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:59.251 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.423 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:59.264 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.427 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:59.267 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.454 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:59.250 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.459 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:59.255 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.463 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:59.303 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.547 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:06:59.343 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.556 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:06:59.396 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:16.635 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:06:59.475 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:24.021 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] read_operations: Running main() from gmock_main.cc 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [----------] Global test environment set-up. 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (42036 ms) 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (58121 ms) 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (6820 ms) 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (71444 ms) 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-25T02:06:24.022 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (34659 ms) 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (16564 ms) 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (11352 ms) 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (5891 ms) 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (4119 ms) 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (5995 ms) 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (10317 ms) 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-25T02:06:24.023 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (14140 ms) 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (4075 ms) 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (4255 ms) 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (289789 ms total) 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [----------] Global test environment tear-down 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (289789 ms total) 2024-03-25T02:06:24.024 INFO:tasks.workunit.client.0.smithi083.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (44003 ms) 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-25T02:06:31.178 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (5193 ms) 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (17904 ms) 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (35711 ms) 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (102811 ms total) 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (11794 ms) 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (33414 ms) 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (77859 ms) 2024-03-25T02:06:31.179 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (123067 ms total) 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (10155 ms) 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (4059 ms) 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (7074 ms) 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (8195 ms) 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (29483 ms total) 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (12542 ms) 2024-03-25T02:06:31.180 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-25T02:06:31.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (8841 ms) 2024-03-25T02:06:31.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (21383 ms total) 2024-03-25T02:06:31.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: 2024-03-25T02:06:31.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-25T02:06:31.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (297281 ms total) 2024-03-25T02:06:31.181 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-25T02:06:36.269 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:19.110 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.327 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:19.123 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.329 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:19.170 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.330 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:19.128 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.395 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:19.192 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.400 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:19.241 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.457 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:19.298 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.463 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:19.303 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:36.538 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:19.379 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:47.966 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: Running main() from gmock_main.cc 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [----------] Global test environment set-up. 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (50967 ms) 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (21405 ms) 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (15146 ms) 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (56306 ms) 2024-03-25T02:06:47.967 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (77604 ms) 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (221428 ms total) 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (18505 ms) 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (17004 ms) 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (16187 ms) 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (10311 ms) 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-25T02:06:47.968 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (6345 ms) 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi083-41871-11. 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: Waiting for snaps to purge. 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (23947 ms) 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (92299 ms total) 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [----------] Global test environment tear-down 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (313727 ms total) 2024-03-25T02:06:47.969 INFO:tasks.workunit.client.0.smithi083.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-25T02:06:52.397 INFO:tasks.workunit.client.0.smithi083.stdout: LibRadosSnapshotsECPP.SnapRemovePP (2109 ms) 2024-03-25T02:06:52.397 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-25T02:06:52.397 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (6178 ms) 2024-03-25T02:06:52.397 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-25T02:06:52.397 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (3246 ms) 2024-03-25T02:06:52.397 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (17748 ms total) 2024-03-25T02:06:52.397 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-03-25T02:06:52.398 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-25T02:06:52.398 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-25T02:06:52.398 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5363 ms) 2024-03-25T02:06:52.398 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-25T02:06:52.398 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5912 ms) 2024-03-25T02:06:52.398 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-25T02:06:52.398 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (6660 ms) 2024-03-25T02:06:52.399 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (17935 ms total) 2024-03-25T02:06:52.399 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: 2024-03-25T02:06:52.399 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-25T02:06:52.399 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (318528 ms total) 2024-03-25T02:06:52.399 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-25T02:06:52.399 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-25T02:06:52.399 INFO:tasks.workunit.client.0.smithi083.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-25T02:06:56.765 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:39.605 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:56.825 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:39.665 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:56.888 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:39.685 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:56.891 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:39.689 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:56.899 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:39.740 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:56.927 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:39.768 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:56.928 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:07:39.770 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:56.952 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:07:39.749 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:06:57.009 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:07:39.850 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.691 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:04.532 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.696 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:04.537 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.738 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:04.579 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.744 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:04.585 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.745 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:04.542 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.748 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:04.589 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.749 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:04.546 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.789 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:04.587 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:21.817 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:04.658 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:36.937 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:07:36.932+0000 12a5a640 -1 osd.1 256 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.21232.0:320 119.1a 119:5b86ab94:::test_313:head [cmpext 0~0 in=4b,write 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected+returnvec+supports_pool_eio e249) 2024-03-25T02:07:41.913 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:24.754 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:41.972 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:24.814 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:42.013 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:24.855 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:42.037 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:24.878 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:42.042 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:24.884 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:42.058 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:24.856 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:42.061 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:24.858 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:42.084 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:24.925 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:42.105 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:24.903 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:49.106 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-25T02:07:49.102+0000 11fa9640 -1 mon.a@0(leader).osd e268 preprocess_pg_ready_to_merge pool for 8.1c dne 2024-03-25T02:07:51.497 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:34.338 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.508 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:34.350 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.560 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:34.402 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.579 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:34.421 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.581 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:34.379 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.588 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:34.429 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.589 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:34.387 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.638 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:34.436 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:07:51.645 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:34.486 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.218 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:45.016 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.221 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:45.019 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.231 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:45.072 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.237 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:45.079 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.281 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:08:45.078 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.310 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:45.152 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.315 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:45.156 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.327 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:08:45.168 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:02.352 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:08:45.194 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:08.899 INFO:tasks.workunit.client.0.smithi083.stdout:pi_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=12 expected=12 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=8 expected=8 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=5 expected=5 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-03-25T02:08:08.900 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=11 expected=11 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:863748b0:::15:head 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=15 expected=15 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=0 expected=0 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=14 expected=14 2024-03-25T02:08:08.901 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: : entry=7 expected=7 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (30435 ms) 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (152450 ms) 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-03-25T02:08:08.902 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (54572 ms) 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 7 tests from LibRadosList (371819 ms total) 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (4273 ms) 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo2,foo3 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo2 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo3 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo1,foo4,foo5 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo4 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo5 2024-03-25T02:08:08.903 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo1 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset foo6,foo7 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo7 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: foo6 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo4 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo5 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo7 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns2:foo6 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: ns1:foo1 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo1 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo2 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: :foo3 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (7497 ms) 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-25T02:08:08.904 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 1 0 2024-03-25T02:08:08.905 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 10 0 2024-03-25T02:08:08.905 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 13 0 2024-03-25T02:08:14.461 INFO:tasks.workunit.client.0.smithi083.stdout: list: Running main() from gmock_main.cc 2024-03-25T02:08:14.461 INFO:tasks.workunit.client.0.smithi083.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-25T02:08:14.461 INFO:tasks.workunit.client.0.smithi083.stdout: list: [----------] Global test environment set-up. 2024-03-25T02:08:14.461 INFO:tasks.workunit.client.0.smithi083.stdout: list: [----------] 3 tests from NeoradosList 2024-03-25T02:08:14.461 INFO:tasks.workunit.client.0.smithi083.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [ OK ] NeoradosList.ListObjects (46494 ms) 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [ OK ] NeoradosList.ListObjectsNS (137229 ms) 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [ OK ] NeoradosList.ListObjectsMany (216553 ms) 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [----------] 3 tests from NeoradosList (400276 ms total) 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [----------] Global test environment tear-down 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [==========] 3 tests from 1 test suite ran. (400276 ms total) 2024-03-25T02:08:14.462 INFO:tasks.workunit.client.0.smithi083.stdout: list: [ PASSED ] 3 tests. 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-03-25T02:08:17.133 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: seed 40920 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (51 ms) 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (30119 ms) 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (443 ms) 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (1139 ms) 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (2005 ms) 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (99 ms) 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-25T02:08:17.134 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (54 ms) 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (176 ms) 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (74 ms) 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (208 ms) 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (62 ms) 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (153 ms) 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (158 ms) 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-25T02:08:17.135 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (255995 ms) 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (11653 ms) 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: done waiting, doing copies 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: done waiting 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (89462 ms) 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (868 ms) 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (47 ms) 2024-03-25T02:08:17.136 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-25T02:08:17.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5477 ms) 2024-03-25T02:08:17.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-25T02:08:17.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-25T02:08:17.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-25T02:08:17.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-25T02:08:17.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-25T02:08:17.137 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 22 misc: Running main() from gmock_main.cc 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [----------] Global test environment set-up. 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.Version (4284 ms) 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-25T02:08:21.847 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4314 ms) 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.LongName (355151 ms) 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (7525 ms) 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (4651 ms) 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (4637 ms) 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.Exec (4276 ms) 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (5563 ms) 2024-03-25T02:08:21.848 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (4350 ms) 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.BigObject (4428 ms) 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2265 ms) 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (6188 ms) 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [----------] 12 tests from NeoRadosMisc (407632 ms total) 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [----------] Global test environment tear-down 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [==========] 12 tests from 1 test suite ran. (407632 ms total) 2024-03-25T02:08:21.849 INFO:tasks.workunit.client.0.smithi083.stdout: misc: [ PASSED ] 12 tests. 2024-03-25T02:08:22.076 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:04.918 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.083 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:04.925 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.148 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:04.946 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.156 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:04.954 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.162 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:05.004 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.166 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:05.008 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.179 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:05.021 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.227 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:05.025 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:22.240 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:05.082 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 7 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 14 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 0 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 15 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 11 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 5 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 8 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 6 0 2024-03-25T02:08:25.246 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 3 0 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 4 0 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 12 0 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 9 0 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2 0 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (14324 ms) 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 3 tests from LibRadosListEC (26094 ms total) 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3314 ms) 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] 1 test from LibRadosListNP (3314 ms total) 2024-03-25T02:08:25.247 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: 2024-03-25T02:08:25.248 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [----------] Global test environment tear-down 2024-03-25T02:08:25.248 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [==========] 11 tests from 3 test suites ran. (411426 ms total) 2024-03-25T02:08:25.248 INFO:tasks.workunit.client.0.smithi083.stdout: api_list: [ PASSED ] 11 tests. 2024-03-25T02:08:42.254 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:25.096 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.258 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:25.100 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.259 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:25.057 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.269 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:25.068 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.309 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:25.151 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.313 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:25.155 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.330 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:25.128 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.332 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:25.174 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:42.374 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:25.216 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: tests from LibRadosMiscPP (398243 ms total) 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (7091 ms) 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (7091 ms total) 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (796 ms) 2024-03-25T02:08:53.937 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (91 ms) 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (887 ms total) 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (813 ms) 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (54 ms) 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (867 ms total) 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-25T02:08:53.938 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (649 ms) 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (301 ms) 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (950 ms total) 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (2020 ms) 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (2020 ms total) 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (440086 ms total) 2024-03-25T02:08:53.939 INFO:tasks.workunit.client.0.smithi083.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-25T02:09:00.909 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: Running main() from gmock_main.cc 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [----------] Global test environment set-up. 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (71234 ms) 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (121932 ms) 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (253550 ms) 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [----------] 3 tests from NeoradosECList (446716 ms total) 2024-03-25T02:09:00.910 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: 2024-03-25T02:09:00.911 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [----------] Global test environment tear-down 2024-03-25T02:09:00.911 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (446716 ms total) 2024-03-25T02:09:00.911 INFO:tasks.workunit.client.0.smithi083.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-25T02:09:02.370 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:45.212 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.372 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:45.214 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.424 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:45.222 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.425 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:09:45.268 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.426 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:45.224 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.445 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:45.287 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.448 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:45.290 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.481 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:09:45.280 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:02.499 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:09:45.341 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.538 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:05.380 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.544 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:05.386 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.603 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:05.401 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.609 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:05.451 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.622 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:05.421 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.633 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:05.476 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.696 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:05.495 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.702 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:05.544 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:22.742 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:05.584 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: Running main() from gmock_main.cc 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [----------] Global test environment set-up. 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (42019 ms) 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-25T02:09:23.205 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (83132 ms) 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (22355 ms) 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (47181 ms) 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (25687 ms) 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (10275 ms) 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (238311 ms) 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (468960 ms total) 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: 2024-03-25T02:09:23.206 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [----------] Global test environment tear-down 2024-03-25T02:09:23.207 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (468960 ms total) 2024-03-25T02:09:23.207 INFO:tasks.workunit.client.0.smithi083.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (28330 ms) 2024-03-25T02:09:23.429 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (61809 ms) 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (22951 ms) 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (89284 ms) 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (21906 ms) 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (6573 ms) 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (238559 ms) 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (469413 ms total) 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: 2024-03-25T02:09:23.430 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-25T02:09:23.431 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (469413 ms total) 2024-03-25T02:09:23.431 INFO:tasks.workunit.client.0.smithi083.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-25T02:09:29.040 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4423 ms) 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (371961 ms) 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8915 ms) 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (5506 ms) 2024-03-25T02:09:29.041 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (4489 ms) 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (4348 ms) 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3378 ms) 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (4077 ms) 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (4404 ms) 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (4523 ms) 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3751 ms) 2024-03-25T02:09:29.042 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3502 ms) 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3372 ms) 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4594 ms) 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3367 ms) 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3411 ms) 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3463 ms) 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3524 ms) 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-25T02:09:29.043 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3455 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3317 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3257 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3398 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2527 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3486 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3459 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3989 ms) 2024-03-25T02:09:29.044 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-25T02:09:30.113 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAi watch_notify: Running main() from gmock_main.cc 2024-03-25T02:09:30.113 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-25T02:09:30.113 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-25T02:09:30.113 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-25T02:09:30.113 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-25T02:09:30.114 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: handle_notify cookie 94459607424176 notify_id 1511828488192 notifier_gid 4457 2024-03-25T02:09:30.114 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (467448 ms) 2024-03-25T02:09:30.114 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-25T02:09:30.114 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: Trying... 2024-03-25T02:09:30.114 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: handle_notify cookie 94459620862848 notify_id 1529008357383 notifier_gid 21571 2024-03-25T02:09:30.114 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: Waiting for 3.000000000s 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: Timed out. 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: Flushing... 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: Flushed... 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (8417 ms) 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (475865 ms total) 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-25T02:09:30.115 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (475865 ms total) 2024-03-25T02:09:30.116 INFO:tasks.workunit.client.0.smithi083.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-25T02:09:42.714 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:25.513 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.716 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:25.515 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.736 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:25.578 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.738 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:25.580 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.761 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:25.560 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.812 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:25.654 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.816 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:25.659 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.819 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:25.662 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:42.871 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:25.713 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.317 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:35.160 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.320 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:35.163 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.329 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:35.128 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.331 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:35.174 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.334 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:35.177 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.382 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:35.181 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.413 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:35.212 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.425 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:35.267 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:09:52.449 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:35.291 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.772 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:45.614 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.844 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:45.687 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.861 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:45.704 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.891 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:10:45.733 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.907 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:45.706 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.912 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:45.755 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.918 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:45.717 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.959 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:10:45.758 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:02.976 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:10:45.818 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:22.875 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:05.718 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:22.881 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:05.724 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:22.951 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:05.794 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:22.957 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:05.756 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:22.959 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:05.758 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:22.961 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:05.804 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:22.964 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:05.807 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:23.009 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:05.808 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:23.026 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:05.868 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:42.897 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:25.739 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:42.954 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:25.796 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:42.998 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:25.841 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:43.011 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:25.811 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:43.014 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:25.813 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:43.059 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:25.858 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:43.065 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:25.907 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:43.066 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:25.909 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:10:43.128 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:25.971 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:02.909 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:45.752 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:02.912 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:45.756 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:02.954 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:11:45.798 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:02.962 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:45.761 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:02.965 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:45.765 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:02.968 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:45.811 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:02.970 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:45.813 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:03.006 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:11:45.805 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:03.013 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:11:45.856 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.225 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:06.068 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.228 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:06.072 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.283 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:06.126 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.291 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:06.090 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.292 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:06.092 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.298 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:06.141 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.302 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:06.146 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.350 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:06.150 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:23.359 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:06.202 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.226 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:26.069 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.231 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:26.075 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.288 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:26.132 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.291 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:26.135 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.302 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:26.102 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.306 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:26.149 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.306 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:26.105 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.348 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:26.191 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:43.365 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:26.165 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.496 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:28.339 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.553 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:28.396 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.570 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:28.414 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.605 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:28.405 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.606 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:28.406 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.607 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:28.451 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.610 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:28.453 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.623 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:28.423 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:45.634 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:28.478 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.280 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:31.123 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.285 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:31.128 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.298 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:31.141 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.333 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:31.176 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.334 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:31.178 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.341 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:31.141 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.343 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:31.143 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.347 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:31.191 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:11:48.356 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:31.156 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.416 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:46.260 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.419 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:46.263 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.433 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:46.233 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.441 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:46.241 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.482 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:12:46.326 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.486 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:12:46.286 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.501 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:46.344 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.511 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:46.355 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:03.565 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:12:46.409 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout:o.RoundTripAppendPP (3408 ms) 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3287 ms) 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2403 ms) 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3076 ms) 2024-03-25T02:12:05.496 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: setting pool EIO 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: max_success 0, min_failed 1 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (5925 ms) 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4043 ms) 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (494038 ms total) 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3450 ms) 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8977 ms) 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (17617 ms) 2024-03-25T02:12:05.497 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3319 ms) 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (33363 ms total) 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2832 ms) 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2832 ms total) 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13880 ms) 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (8128 ms) 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-25T02:12:05.498 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7146 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6885 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3751 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7576 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7107 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7836 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7056 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6955 ms) 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-25T02:12:05.499 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7123 ms) 2024-03-25T02:12:05.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-25T02:12:05.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6812 ms) 2024-03-25T02:12:05.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-25T02:12:05.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7801 ms) 2024-03-25T02:12:05.500 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-03-25T02:12:18.326 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:01.169 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.330 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:01.174 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.336 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:01.136 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.338 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:01.138 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.364 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:01.208 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.374 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:01.175 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.383 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:01.227 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.387 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:01.231 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:18.420 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:01.263 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.464 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:11.308 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.469 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:11.313 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.500 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:11.344 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.523 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:11.367 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.524 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:11.368 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.534 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:11.335 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.539 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:11.340 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.559 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:11.403 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:28.569 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:11.370 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.540 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:16.383 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.548 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:16.348 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.549 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:16.392 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.553 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:16.354 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.572 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:16.372 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.583 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:16.427 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.599 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:16.443 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.608 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:16.452 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:33.628 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:16.472 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.607 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:31.451 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.648 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:31.491 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.680 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:31.480 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.681 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:31.482 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.683 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:31.527 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.687 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:31.531 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.713 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:31.557 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.723 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:31.523 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:48.728 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:31.572 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3472 ms) 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7687 ms) 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (8067 ms) 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7310 ms) 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6960 ms) 2024-03-25T02:12:49.635 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7068 ms) 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7044 ms) 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (145664 ms total) 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (675897 ms total) 2024-03-25T02:12:49.636 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-25T02:13:03.523 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:46.367 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.529 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:46.372 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.564 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:46.409 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.577 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:46.377 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.582 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:46.426 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.583 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:46.428 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.585 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:46.386 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.614 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:46.415 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:03.616 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:46.460 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.677 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:51.521 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.679 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:51.523 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.696 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:13:51.541 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.731 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:51.575 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.732 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:51.576 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.748 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:13:51.592 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.749 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:51.550 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.753 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:51.553 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:08.768 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:13:51.568 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.727 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:01.571 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.731 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:01.576 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.757 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:01.602 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.778 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:01.579 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.780 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:01.624 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.781 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:01.582 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.784 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:01.629 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.807 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:01.608 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:18.811 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:01.655 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.611 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:11.455 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.642 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:11.486 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.664 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:11.508 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.665 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:11.509 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.666 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:11.510 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.669 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:11.469 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.670 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:11.470 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.689 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:11.533 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:28.704 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:11.504 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: seed 40944 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (26143 ms) 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (187070 ms) 2024-03-25T02:13:32.986 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (786 ms) 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (213999 ms total) 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (47617 ms) 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (26326 ms) 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (23686 ms) 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [3] 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [4,3] 2024-03-25T02:13:32.987 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting some heads 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: waiting for scrubs... 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: done waiting 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (211226 ms) 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (11882 ms) 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (9895 ms) 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-25T02:13:32.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (10041 ms) 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10271 ms) 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12162 ms) 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10880 ms) 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10126 ms) 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (48142 ms) 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (11998 ms) 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-25T02:13:32.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11987 ms) 2024-03-25T02:13:32.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-25T02:13:32.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16490 ms) 2024-03-25T02:13:32.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-25T02:13:32.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8944 ms) 2024-03-25T02:13:32.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-25T02:13:32.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (13017 ms) 2024-03-25T02:13:48.745 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:31.589 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.804 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:31.605 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.809 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:31.610 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.812 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:31.656 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.872 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:31.716 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.873 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:31.717 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.876 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:31.677 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.878 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:31.722 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:48.930 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:31.774 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:13:54.852 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-25T02:13:54.852 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-25T02:13:54.852 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: ok, hit_set contains 242:602f83fe:::foo:head 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9090 ms) 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg_num = 32 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 0 ls 1711332829,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 1 ls 1711332830,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 2 ls 1711332830,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 3 ls 1711332829,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 4 ls 1711332829,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 5 ls 1711332829,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 6 ls 1711332830,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 7 ls 1711332830,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 8 ls 1711332829,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 9 ls 1711332829,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 10 ls 1711332829,0 2024-03-25T02:13:54.853 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 11 ls 1711332829,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 12 ls 1711332829,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 13 ls 1711332830,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 14 ls 1711332830,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 15 ls 1711332830,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 16 ls 1711332830,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 17 ls 1711332829,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 18 ls 1711332829,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 19 ls 1711332829,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 20 ls 1711332829,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 21 ls 1711332830,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 22 ls 1711332829,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 23 ls 1711332830,0 2024-03-25T02:13:54.854 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 24 ls 1711332829,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 25 ls 1711332829,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 26 ls 1711332829,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 27 ls 1711332830,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 28 ls 1711332830,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 29 ls 1711332830,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 30 ls 1711332830,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg 31 ls 1711332829,0 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: pg_num = 32 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6cac518f:::0:head 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:02547ec2:::1:head 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f905c69b:::2:head 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:cfc208b3:::3:head 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d83876eb:::4:head 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b29083e3:::5:head 2024-03-25T02:13:54.855 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c4fdafeb:::6:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:5c6b0b28:::7:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:bd63b0f1:::8:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:e960b815:::9:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:52ea6a34:::10:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:89d3ae78:::11:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:de5d7c5f:::12:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:566253c9:::13:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:62a1935d:::14:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:863748b0:::15:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:3958e169:::16:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:4d4dabf9:::17:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:8391935d:::18:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:28883081:::19:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:69259c59:::20:head 2024-03-25T02:13:54.856 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:4bdb80b7:::21:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a11c5d71:::22:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:271af37b:::23:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:95b121be:::24:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:58d1031b:::25:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:0a050783:::26:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c709704c:::27:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:cbe56eaf:::28:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:86b4b162:::29:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:70d89383:::30:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:dd450c7c:::31:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6d5729b1:::32:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c388f3fb:::33:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:56cfea31:::34:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:9dbc1bf7:::35:head 2024-03-25T02:13:54.857 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:40b74ccd:::36:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:4d5aaf42:::37:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:920f362c:::38:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6cc53222:::39:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:9cad833f:::40:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:1ea84d41:::41:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c4480ef6:::42:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a694361e:::43:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d1bd33e9:::44:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:ddc2cd5d:::45:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:2b782207:::46:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:7b187fca:::47:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:90ecdf6f:::48:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a5ed95fe:::49:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:ea0eaa55:::50:head 2024-03-25T02:13:54.858 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f33ef17b:::51:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a0d1b2f6:::52:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:60c5229e:::53:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:edcbc575:::54:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:102cf253:::55:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:efb7fb0b:::56:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:50d0a326:::57:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d4dc5daf:::58:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:3a130462:::59:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:ec87ed71:::60:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d5bc9454:::61:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:3ddfe313:::62:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:7c2816b9:::63:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:47e00e4d:::64:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c6410c18:::65:head 2024-03-25T02:13:54.859 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b48ed237:::66:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:cd63ad31:::67:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b179e92b:::68:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:0d9f741a:::69:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6d3352ae:::70:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c6d5c19e:::71:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:bc4729c3:::72:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:77e930b9:::73:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:0abeecfd:::74:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b7c37e15:::75:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b6378398:::76:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:02bd68de:::77:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:cc795d2d:::78:head 2024-03-25T02:13:54.860 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:630d4fea:::79:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:e0d29ef5:::80:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:fd6f13d2:::81:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:606461d5:::82:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:eadbdc43:::83:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:8761d0bb:::84:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:9ef0186f:::85:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:e0d41294:::86:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:961de695:::87:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:1423148f:::88:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:633a8fa2:::89:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a8653809:::90:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:3dac8b33:::91:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:35aad435:::92:head 2024-03-25T02:13:54.861 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f6dcc343:::93:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:dbbdad87:::94:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:1cb48ce0:::95:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:03cd461c:::96:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:17a4ea99:::97:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:9993c9a7:::98:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6394211c:::99:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:94c7ae57:::100:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6fdee5bb:::101:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:9a477fd1:::102:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:eb850916:::103:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:affc56b9:::104:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b42dc814:::105:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f319f8f0:::106:head 2024-03-25T02:13:54.862 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:9a40b9de:::107:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:8b524f28:::108:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:e3de589f:::109:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:90f90a5b:::110:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a7b4f1d7:::111:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:af51766e:::112:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b6f90bd1:::113:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:e0261208:::114:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c9569ef7:::115:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:61bebe50:::116:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:fe93412b:::117:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d3d38bee:::118:head 2024-03-25T02:13:54.863 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:3100ba0c:::119:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d0560ada:::120:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f0ea8b35:::121:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:766f231a:::122:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a07a2582:::123:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:bd7c6b3a:::124:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:fb2ddaff:::125:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:4408e1fe:::126:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:ee1df7a7:::127:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c3002909:::128:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:4f48ffa9:::129:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:edf38733:::130:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c08425c0:::131:head 2024-03-25T02:13:54.864 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:5f902d98:::132:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:41ea2c93:::133:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:813cee13:::134:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:0131818d:::135:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:26ba5a85:::136:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:381b8a5a:::137:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:28797e47:::138:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:bfca7f22:::139:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:36807075:::140:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:80b03975:::141:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:5c15709b:::142:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f39ea15e:::143:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:ea992956:::144:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:48887b1c:::145:head 2024-03-25T02:13:54.865 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:9f24a9dd:::146:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:987f100b:::147:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d2dd3581:::148:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:7fed1808:::149:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c80b70e9:::150:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:85ed90f9:::151:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:36428b24:::152:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d044c34a:::153:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:7c18bf58:::154:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d1c21232:::155:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a7a3c575:::156:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:87da0633:::157:head 2024-03-25T02:13:54.866 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d5ac3822:::158:head 2024-03-25T02:13:54.867 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:3f20522d:::159:head 2024-03-25T02:13:54.867 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6ca26563:::160:head 2024-03-25T02:13:54.867 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:532ce135:::161:head 2024-03-25T02:13:54.867 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:c78863e6:::162:head 2024-03-25T02:14:03.750 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:46.594 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.754 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:46.598 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.778 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:46.623 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.803 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:46.604 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.804 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:46.649 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.805 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:46.605 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.806 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:46.651 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.825 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:46.625 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:03.829 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:46.673 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.761 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:51.605 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.763 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:51.608 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.779 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:14:51.624 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.810 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:51.611 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.811 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:51.612 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.825 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:14:51.626 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.827 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:51.672 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.829 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:51.673 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:08.843 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:14:51.687 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.830 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:06.674 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.832 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:06.677 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.854 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:06.699 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.878 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:06.679 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.881 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:06.682 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.882 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:06.726 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.883 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:06.728 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.902 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:06.703 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:23.905 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:06.750 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:28.927 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:11.772 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:28.985 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:11.829 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:28.989 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:11.790 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:28.993 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:11.794 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.013 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:11.857 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.018 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:11.819 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.061 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:11.906 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.065 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:11.909 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.088 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:11.933 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.519 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:12.320 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.520 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:12.321 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.529 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:12.330 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.673 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:12.518 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.674 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:12.519 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.684 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:12.529 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.687 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:12.531 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.688 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:12.533 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:29.778 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:12.623 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.106 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:21.951 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.109 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:21.954 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.140 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:21.984 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.166 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:21.967 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.168 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:21.969 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.175 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:22.020 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.178 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:22.023 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.198 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:21.999 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:39.211 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:22.055 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.058 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:31.903 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.060 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:31.905 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.084 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:31.929 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.107 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:31.908 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.108 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:31.910 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.110 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:31.954 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.111 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:31.956 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.133 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:31.934 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:49.147 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:31.992 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.074 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:36.919 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.075 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:36.876 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.076 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:36.921 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.078 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:36.879 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.090 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:36.891 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.095 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:36.940 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.129 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:36.974 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.131 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:36.975 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:14:54.148 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:36.993 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:06.160 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:7462ddf6:::163:head 2024-03-25T02:15:06.160 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:5d165639:::164:head 2024-03-25T02:15:06.160 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f43765fc:::165:head 2024-03-25T02:15:06.160 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b4c720e9:::166:head 2024-03-25T02:15:06.160 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:e694b040:::167:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:afa38db2:::168:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:77ba9f53:::169:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:87495034:::170:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:7c96bf0e:::171:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:dbe346cc:::172:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:e943ec24:::173:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f97a9c0c:::174:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:6f26e74d:::175:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:4f95e106:::176:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:0e6f2f8f:::177:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:05db05f1:::178:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:38a78d66:::179:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:d095610b:::180:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:a1a9d709:::181:head 2024-03-25T02:15:06.161 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:1e5d39db:::182:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:f7df4fb9:::183:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:03a7f161:::184:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:ba70721e:::185:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:28e5662d:::186:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:973d52de:::187:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:4303eb1c:::188:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:b990b48e:::189:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:29b8165b:::190:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:3547f197:::191:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:7e260936:::192:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:1abec7b1:::193:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:10fdda93:::194:head 2024-03-25T02:15:06.162 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:15817eea:::195:head 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:770bab57:::196:head 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:ed9e13e7:::197:head 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:71471a8f:::198:head 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: checking for 243:10fb1d02:::199:head 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22818 ms) 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,0 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first is 1711332853 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,0 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,0 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,0 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,0 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,0 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,1711332859,1711332860,0 2024-03-25T02:15:06.163 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,1711332859,1711332860,0 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,1711332859,1711332860,0 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,1711332859,1711332860,1711332862,1711332863,0 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,1711332859,1711332860,1711332862,1711332863,0 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332853,1711332856,1711332857,1711332859,1711332860,1711332862,1711332863,0 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711332857,1711332859,1711332860,1711332862,1711332863,1711332865,1711332866,0 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first now 1711332857, trimmed 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (23383 ms) 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: foo0 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (17372 ms) 2024-03-25T02:15:06.164 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-25T02:15:14.267 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:57.112 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.271 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:57.116 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.301 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:15:57.146 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.321 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:57.123 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.323 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:57.124 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.345 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:57.190 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.347 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:57.192 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.350 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:15:57.151 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:14.375 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:15:57.220 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.237 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:12.082 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.240 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:12.085 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.265 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:12.110 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.290 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:12.092 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.292 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:12.094 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.307 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:12.152 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.308 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:12.154 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.323 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:12.124 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:29.336 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:12.181 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.331 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:17.176 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.333 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:17.134 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.335 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:17.136 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.336 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:17.182 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.339 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:17.184 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.341 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:17.186 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.400 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:17.201 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.415 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:17.260 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:34.423 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:17.268 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.327 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:32.172 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.329 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:32.174 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.358 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:32.203 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.378 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:32.223 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.380 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:32.225 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.380 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:32.182 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.381 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:32.183 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.419 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:32.264 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:49.423 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:32.225 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.401 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:37.246 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.408 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:37.210 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.409 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:37.255 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.416 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:37.261 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.416 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:37.217 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.417 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:37.262 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.495 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:37.296 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.504 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:37.349 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:15:54.510 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:37.355 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.393 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:47.238 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.398 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:47.243 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.423 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:47.269 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.448 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:47.249 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.449 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:47.251 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.458 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:47.303 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.459 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:47.305 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.477 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:47.278 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:04.485 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:47.330 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.486 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:57.331 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.545 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:57.346 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.546 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:57.348 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.560 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:16:57.362 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.566 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:57.412 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.588 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:16:57.433 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.631 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:57.477 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.634 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:57.479 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:14.664 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:16:57.510 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.879 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:01.724 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.890 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:01.735 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.891 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:01.692 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.926 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:01.772 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.935 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:01.781 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.936 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:01.737 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.946 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:01.792 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.956 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:01.758 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:18.961 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:01.806 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.516 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:07.362 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.580 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:07.425 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.584 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:07.386 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.587 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:07.389 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.596 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:07.442 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.599 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:07.401 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.654 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:07.499 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.664 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:07.510 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:24.677 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:07.522 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.679 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:17.525 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.682 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:17.527 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.698 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:17.544 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.738 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:17.539 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.742 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:17.587 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.742 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:17.544 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.746 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:17.591 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.760 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:17.561 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:34.768 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:17.613 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.536 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:27.381 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.537 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:27.383 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.539 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:27.340 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.542 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:27.344 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.561 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:27.406 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.566 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:27.366 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.589 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:27.434 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.590 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:27.435 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:44.615 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:27.460 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.669 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:37.514 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.671 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:37.517 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.691 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:37.536 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.728 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:37.530 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.729 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:37.531 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.732 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:37.578 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.739 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:37.584 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.750 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:37.552 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:54.761 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:37.607 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.684 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:42.530 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.686 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:42.532 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.705 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:42.551 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.736 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:42.581 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.737 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:42.583 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.750 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:42.552 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.753 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:42.598 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.754 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:42.556 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:16:59.768 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:42.570 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.782 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:47.628 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.843 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:47.689 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.850 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:47.651 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.853 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:47.654 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.881 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:47.683 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.884 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:47.730 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.916 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:47.762 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.919 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:47.764 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:04.944 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:47.790 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.766 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:57.612 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.817 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:57.662 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.818 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:57.620 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.821 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:57.623 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.838 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:57.684 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.843 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:57.645 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.866 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:57.711 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.869 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:57.715 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:14.889 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:57.734 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.454 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:58.256 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.455 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:58.257 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.463 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:17:58.265 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.482 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:58.328 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.483 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:58.329 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.492 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:17:58.338 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.513 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:58.359 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.514 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:58.360 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:15.527 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:17:58.372 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.211 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:18.056 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.218 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:18.064 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.249 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:18.094 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.283 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:18.085 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.293 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:18.095 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.298 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:18.144 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.306 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:18.151 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.328 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:18.130 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:35.344 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:18.190 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.140 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:37.985 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.142 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:37.988 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.175 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:38.021 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.206 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:38.008 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.207 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:38.009 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.208 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:38.053 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.211 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:38.057 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.239 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:38.041 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:17:55.239 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:38.085 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.055 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:52.901 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.058 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:52.904 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.085 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:52.930 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.108 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:52.910 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.109 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:52.911 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.121 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:52.967 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.122 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:52.968 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.135 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:52.937 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:10.151 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:52.996 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.023 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:57.869 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.073 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:57.875 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.074 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:57.876 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.078 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:57.924 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.087 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:18:57.889 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.094 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:18:57.940 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.148 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:57.993 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.151 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:57.997 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:15.165 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:18:58.011 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:25.313 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:18:25.308+0000 25281640 -1 snap_mapper.add_oid found existing snaps mapped on 261:abe8793c:test-rados-api-smithi083-40944-80::foo:2, removing 2024-03-25T02:18:25.321 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:18:25.323+0000 2a28f640 -1 snap_mapper.add_oid found existing snaps mapped on 261:abe8793c:test-rados-api-smithi083-40944-80::foo:2, removing 2024-03-25T02:18:30.135 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:12.980 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.139 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:12.985 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.161 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:13.007 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.190 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:12.992 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.192 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:12.994 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.192 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:13.038 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.194 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:13.040 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.216 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:13.018 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:30.218 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:13.064 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.150 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:17.996 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.154 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:18.000 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.168 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:18.014 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.200 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:18.002 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.201 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:18.004 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.210 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:18.056 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.211 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:18.057 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.215 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:18.017 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:35.225 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:18.071 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.167 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:28.013 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.172 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:28.018 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.195 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:28.041 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.217 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:28.019 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.218 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:28.021 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.226 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:28.072 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.228 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:28.074 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.241 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:28.043 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:45.253 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:28.099 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.899 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:38.745 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.911 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:38.757 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.912 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:38.758 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.914 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:38.716 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.915 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:38.761 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.916 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:38.719 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.992 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:38.838 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:55.992 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:38.838 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:18:56.015 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:38.817 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.211 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:43.057 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.213 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:43.059 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.225 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:43.072 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.261 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:43.064 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.262 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:43.065 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.263 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:43.109 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.265 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:43.111 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.276 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:43.078 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.277 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:43.124 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.893 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:43.739 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.894 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:43.740 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.895 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:43.697 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.896 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:43.699 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.903 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:43.749 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.904 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:43.707 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.936 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:43.782 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.937 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:43.783 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:00.945 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:43.791 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20512 ms) 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (25063 ms) 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4276 ms) 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5397 ms) 2024-03-25T02:19:01.594 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4537 ms) 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5644 ms) 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5589 ms) 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39909 ms) 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18035 ms) 2024-03-25T02:19:01.595 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5025 ms) 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4406 ms) 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (28558 ms) 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (17008 ms) 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5466 ms) 2024-03-25T02:19:01.596 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5038 ms) 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: waiting for scrubs... 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: done waiting 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26319 ms) 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12504 ms) 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11327 ms) 2024-03-25T02:19:01.597 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-25T02:19:01.598 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:19:01.598 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11334 ms) 2024-03-25T02:19:15.239 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:58.085 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.244 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:58.091 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.276 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:19:58.122 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.292 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:58.094 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.293 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:58.095 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.295 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:58.141 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.296 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:58.143 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.323 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:19:58.125 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:15.346 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:19:58.192 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.304 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:03.150 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.310 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:03.112 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.311 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:03.157 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.312 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:03.158 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.313 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:03.159 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.356 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:03.159 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.372 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:03.174 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.375 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:03.221 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:20.386 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:03.232 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.482 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:13.328 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.514 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:13.360 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.553 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:13.356 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.556 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:13.358 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.556 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:13.401 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.577 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:13.380 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.603 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:13.450 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.607 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:13.454 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:30.630 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:13.476 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.297 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:23.144 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.342 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:23.188 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.359 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:23.206 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.393 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:23.239 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.394 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:23.241 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.405 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:23.207 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.409 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:23.209 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.410 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:23.256 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:40.425 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:23.228 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.524 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:33.371 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.527 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:33.373 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.546 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:33.393 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.581 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:33.383 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.584 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:33.386 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.586 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:33.432 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.590 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:33.436 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.599 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:33.402 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:50.608 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:33.455 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.468 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:38.314 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.471 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:38.318 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.493 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:38.339 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.530 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:38.332 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.531 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:38.334 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.548 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:38.394 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.555 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:38.358 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.574 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:38.420 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:19:55.593 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:38.439 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.656 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:43.502 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.660 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:43.507 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.680 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:43.527 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.709 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:43.512 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.710 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:43.513 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.714 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:43.560 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.719 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:43.565 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.725 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:43.528 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:00.733 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:43.580 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.539 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:48.385 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.542 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:48.389 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.556 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:20:48.402 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.585 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:48.387 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.588 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:48.391 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.593 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:48.440 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.595 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:48.442 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.602 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:20:48.404 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:05.605 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:20:48.451 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:08.899 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:20:08.901+0000 25281640 -1 snap_mapper.add_oid found existing snaps mapped on 115:c08c10a0:test-rados-api-smithi083-40944-97::foo:21, removing 2024-03-25T02:20:08.930 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:20:08.931+0000 29289640 -1 snap_mapper.add_oid found existing snaps mapped on 115:c08c10a0:test-rados-api-smithi083-40944-97::foo:21, removing 2024-03-25T02:20:20.834 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:03.680 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.836 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:03.683 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.874 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:03.720 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.886 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:03.689 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.890 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:03.693 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.910 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:03.756 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.916 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:03.763 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.923 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:03.725 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:20.958 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:03.804 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.848 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:18.694 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.850 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:18.697 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.891 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:18.738 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.914 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:18.717 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.916 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:18.719 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.928 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:18.774 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.937 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:18.784 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.953 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:18.755 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:35.981 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:18.827 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.656 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:23.502 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.706 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:23.552 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.706 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:23.552 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.708 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:23.511 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.712 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:23.515 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.713 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:23.560 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.726 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:23.529 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.729 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:23.576 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:40.771 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:23.617 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.746 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:33.592 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.748 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:33.595 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.775 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:33.622 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.799 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:33.645 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.801 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:33.647 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.824 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:33.671 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.825 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:33.627 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.825 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:33.628 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:20:50.847 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:33.650 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.777 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:43.623 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.832 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:43.679 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.859 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:43.705 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.881 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:43.683 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.883 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:43.686 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.904 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:43.750 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.906 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:43.753 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.913 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:43.715 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:00.946 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:43.793 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.733 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:48.579 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.748 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:48.595 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.789 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:48.636 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.797 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:48.643 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.809 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:48.655 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.809 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:48.612 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.820 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:48.622 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.831 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:48.677 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:05.852 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:48.655 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.739 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:49.585 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.740 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:49.587 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.740 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:49.543 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.741 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:49.545 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.749 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:21:49.596 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.751 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:21:49.554 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.982 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:49.829 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.983 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:49.830 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:06.999 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:21:49.846 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:20.960 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:03.807 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:20.964 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:03.811 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:20.999 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:03.846 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.013 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:03.816 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.015 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:03.818 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.017 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:03.863 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.020 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:03.866 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.054 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:03.901 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.055 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:03.857 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.777 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:04.580 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.778 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:04.581 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.788 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:04.591 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.853 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:04.699 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.854 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:04.701 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.855 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:04.702 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.857 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:04.703 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.869 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:04.713 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:21.870 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:04.717 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.093 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:18.940 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.153 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:19.000 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.195 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:19.042 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.203 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:19.006 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.208 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:19.011 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.219 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:19.066 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.220 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:19.067 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.242 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:19.045 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:36.265 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:19.109 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.198 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:24.045 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.199 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:24.046 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.214 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:24.061 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.259 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:24.062 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.264 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:24.067 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.266 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:24.113 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.270 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:24.117 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.279 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:24.082 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:41.290 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:24.135 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.078 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:33.924 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.080 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:33.927 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.103 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:33.950 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.128 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:33.930 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.128 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:33.931 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.132 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:33.979 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.133 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:33.980 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.150 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:33.953 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:21:51.154 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:34.001 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.121 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:43.967 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.125 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:43.972 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.144 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:43.991 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.170 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:43.973 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.170 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:43.974 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.178 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:44.025 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.179 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:44.026 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.182 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:43.986 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:01.191 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:44.038 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.263 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:54.110 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.265 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:54.112 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.292 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:22:54.138 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.328 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:54.174 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.329 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:54.176 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.334 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:54.137 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.340 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:54.143 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.359 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:22:54.206 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:11.364 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:22:54.167 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.345 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:09.191 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.350 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:09.197 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.388 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:09.235 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.404 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:09.250 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.405 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:09.252 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.412 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:09.215 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.414 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:09.217 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.451 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:09.254 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:26.469 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:09.316 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.236 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:24.083 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.304 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:24.151 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.342 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:24.189 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.362 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:24.209 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.364 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:24.211 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.372 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:24.175 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.374 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:24.177 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.401 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:24.248 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:41.413 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:24.215 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.321 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:29.167 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.330 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:29.177 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.344 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:29.191 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.368 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:29.171 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.369 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:29.173 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.372 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:29.219 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.373 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:29.220 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.383 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:29.186 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:46.388 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:29.235 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.469 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:39.316 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.526 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:39.373 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.556 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:39.403 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.582 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:39.429 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.583 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:39.386 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.584 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:39.430 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.586 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:39.390 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.612 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:39.459 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:22:56.613 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:39.415 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:00.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-25T02:23:00.000 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10928 ms) 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (7778 ms) 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6669 ms) 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (28044 ms) 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:23:00.001 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15167 ms) 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12405 ms) 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 00000012 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13081 ms) 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: require_osd_release = squid 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7869 ms) 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (925242 ms total) 2024-03-25T02:23:00.002 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1872 ms) 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10465 ms) 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19019 ms) 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31374 ms total) 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-25T02:23:00.003 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6857 ms) 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8525 ms) 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: waiting for scrub... 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: done waiting 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26547 ms) 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11119 ms) 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9057 ms) 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10105 ms) 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-25T02:23:00.004 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11584 ms) 2024-03-25T02:23:00.005 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-25T02:23:06.514 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:49.361 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.518 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:49.365 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.546 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:49.393 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.562 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:49.366 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.564 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:49.368 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.571 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:49.418 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.572 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:49.420 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.589 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:49.392 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:06.599 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:49.446 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.306 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:50.109 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.307 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:50.111 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.320 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:50.123 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.454 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:50.301 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.455 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:50.302 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.459 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:50.306 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.461 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:50.308 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.468 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:50.315 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:07.481 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:50.328 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.506 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:54.353 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.511 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:54.358 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.530 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:23:54.377 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.564 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:54.367 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.565 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:54.368 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.576 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:54.423 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.577 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:54.424 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.580 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:23:54.383 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:11.596 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:23:54.443 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.477 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:09.324 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.478 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:09.326 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.512 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:09.359 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.526 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:09.329 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.529 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:09.332 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.551 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:09.398 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.552 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:09.399 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.559 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:09.362 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:26.587 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:09.434 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.322 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:10.125 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.323 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:10.126 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.338 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:10.141 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.458 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:10.305 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.459 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:10.306 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.468 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:10.316 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.480 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:10.327 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.481 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:10.328 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:27.490 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:10.337 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.696 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:29.543 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.753 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:29.600 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.758 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:29.561 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.761 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:29.565 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.809 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:29.613 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.815 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:29.662 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.825 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:29.672 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.832 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:29.679 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:23:46.880 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:29.727 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.618 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:44.465 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.622 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:44.469 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.670 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:44.518 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.674 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:44.477 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.675 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:44.479 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.688 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:44.535 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.691 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:44.538 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.713 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:44.516 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:01.725 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:44.573 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.603 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:49.450 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.660 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:49.463 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.662 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:49.509 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.666 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:49.470 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.693 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:49.496 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.693 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:49.541 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.720 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:49.567 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.745 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:49.592 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:06.766 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:49.613 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.666 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:59.512 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.668 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:59.515 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.701 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:24:59.548 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.716 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:59.520 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.717 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:59.521 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.724 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:59.571 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.729 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:59.575 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.738 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:24:59.542 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:16.750 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:24:59.597 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.789 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:09.636 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.794 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:09.641 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.811 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:09.658 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.851 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:09.654 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.856 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:09.660 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.858 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:09.705 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.872 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:09.675 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.880 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:09.727 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:26.898 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:09.745 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.663 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:14.510 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.716 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:14.563 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.722 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:14.526 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.726 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:14.530 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.735 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:14.582 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.740 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:14.544 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.776 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:14.623 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.779 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:14.626 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:31.794 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:14.641 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.776 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:29.623 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.781 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:29.628 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.842 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:29.689 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.850 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:29.654 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.852 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:29.656 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.867 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:29.714 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.869 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:29.716 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.895 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:29.698 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:46.911 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:29.758 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.455 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:30.258 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.456 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:30.259 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.464 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:30.267 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.531 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:30.378 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.532 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:30.379 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.541 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:30.388 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.546 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:30.393 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.547 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:30.395 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:24:47.558 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:30.405 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:06.982 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:49.828 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:06.984 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:49.832 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:07.018 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:49.865 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:07.018 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:49.822 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:07.025 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:49.829 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:07.041 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:49.888 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:07.044 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:49.892 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:07.058 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:49.862 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:07.110 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:49.957 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:16.953 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:59.800 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.023 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:59.870 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.024 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:59.871 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.030 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:59.878 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.056 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:25:59.903 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.070 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:25:59.918 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.074 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:59.877 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.075 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:59.879 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:17.103 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:25:59.907 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.012 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:04.860 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.015 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:04.863 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.029 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:04.877 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.061 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:04.864 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.062 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:04.866 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.065 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:04.912 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.068 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:04.916 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.074 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:04.878 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:22.083 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:04.930 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.880 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:09.727 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.885 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:09.732 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.896 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:09.743 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.936 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:09.739 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.936 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:09.740 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.947 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:09.751 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.954 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:09.801 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.956 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:09.803 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:26.965 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:09.813 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.921 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:19.768 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.925 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:19.773 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.948 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:19.795 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.970 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:19.774 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.972 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:19.775 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.974 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:19.821 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.975 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:19.823 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:36.995 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:19.799 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:37.000 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:19.847 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:46.960 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:29.807 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:46.963 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:29.810 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:46.989 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:29.836 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:47.013 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:29.861 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:47.014 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:29.862 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:47.015 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:29.819 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:47.016 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:29.820 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:47.041 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:29.888 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:47.043 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:29.845 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10517 ms) 2024-03-25T02:25:52.006 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13328 ms) 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (9834 ms) 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15183 ms) 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7691 ms) 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9501 ms) 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-25T02:25:52.007 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: ok, hit_set contains 293:602f83fe:::foo:head 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9827 ms) 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first is 1711333453 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,1711333459,1711333460,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,1711333459,1711333460,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,1711333459,1711333460,0 2024-03-25T02:25:52.008 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,1711333459,1711333460,1711333462,1711333463,0 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,1711333459,1711333460,1711333462,1711333463,0 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333453,1711333456,1711333457,1711333459,1711333460,1711333462,1711333463,0 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: got ls 1711333457,1711333459,1711333460,1711333462,1711333463,1711333465,1711333466,0 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: first now 1711333457, trimmed 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22726 ms) 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: foo0 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (17435 ms) 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20452 ms) 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25242 ms) 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-25T02:25:52.009 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4986 ms) 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4471 ms) 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4808 ms) 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3436 ms) 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (263232 ms total) 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1458142 ms total) 2024-03-25T02:25:52.010 INFO:tasks.workunit.client.0.smithi083.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-25T02:25:52.012 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.012 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40663 2024-03-25T02:25:52.012 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40663 2024-03-25T02:25:52.012 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40863 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40863 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41331 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41331 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41110 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41110 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40972 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40972 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40748 2024-03-25T02:25:52.013 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40748 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41052 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41052 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41501 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41501 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41513 2024-03-25T02:25:52.014 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41513 2024-03-25T02:25:52.171 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:35.019 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.174 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:35.021 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.194 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:35.041 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.226 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:35.030 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.227 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:35.031 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.232 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:35.080 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.236 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:35.084 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.247 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:35.051 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:52.254 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:35.101 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.070 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:39.917 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.074 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:39.922 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.090 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:39.937 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.122 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:39.925 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.123 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:39.927 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.129 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:39.977 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.131 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:39.978 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.134 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:39.938 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:25:57.142 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:39.989 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.090 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:49.937 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.093 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:49.941 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.106 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:49.954 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.142 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:49.946 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.143 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:49.947 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.155 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:49.959 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.156 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:50.004 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.157 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:50.005 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:07.169 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:50.017 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.105 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:54.953 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.111 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:54.958 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.121 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:26:54.968 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.156 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:54.960 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.157 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:54.961 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.164 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:55.011 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.165 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:55.013 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.167 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:26:54.970 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:12.176 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:26:55.023 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.142 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:09.990 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.145 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:09.992 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.158 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:10.005 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.195 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:10.042 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.196 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:10.043 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.198 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:10.002 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.199 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:10.003 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.210 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:10.058 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:27.213 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:10.017 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.119 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:19.967 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.168 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:20.016 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.171 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:19.975 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.174 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:19.978 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.180 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:20.028 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.186 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:19.990 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.216 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:20.064 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.218 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:20.065 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:37.230 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:20.077 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.185 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:30.032 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.189 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:30.037 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.200 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:30.048 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.236 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:30.040 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.237 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:30.084 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.238 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:30.085 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.239 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:30.043 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.250 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:30.054 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:47.250 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:30.098 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.208 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:40.055 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.210 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:40.058 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.227 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:40.075 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.257 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:40.061 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.258 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:40.062 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.264 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:40.111 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.265 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:40.113 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.269 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:40.073 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:26:57.277 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:40.125 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.230 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:50.078 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.234 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:50.082 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.245 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:27:50.092 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.281 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:50.129 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.281 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:50.085 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.282 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:50.130 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.282 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:50.086 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.296 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:27:50.100 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:07.297 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:27:50.145 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.266 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:00.113 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.270 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:00.117 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.281 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:00.129 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.321 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:00.125 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.322 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:00.126 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.333 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:00.137 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.345 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:00.192 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.347 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:00.195 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:17.360 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:00.208 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.289 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:10.137 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.294 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:10.142 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.306 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:10.154 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.341 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:10.145 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.342 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:10.189 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.342 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:10.146 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.345 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:10.192 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.354 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:10.158 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:27.356 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:10.204 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.327 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:20.174 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.331 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:20.179 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.348 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:20.195 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.377 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:20.181 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.379 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:20.183 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.380 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:20.228 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.382 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:20.230 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.392 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:20.196 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:37.395 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:20.243 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.460 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:30.308 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.467 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:30.315 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.488 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:30.336 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.526 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:30.330 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.526 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:30.374 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.529 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:30.334 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.530 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:30.378 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.548 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:30.353 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:47.549 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:30.397 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.213 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:31.061 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.214 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:31.062 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.223 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:31.071 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.236 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:31.084 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.237 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:31.085 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.246 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:31.094 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.247 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:31.051 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.248 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:31.052 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:48.256 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:31.061 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.477 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:40.324 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.481 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:40.329 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.492 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:40.340 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.527 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:40.331 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.528 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:40.333 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.540 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:40.345 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.547 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:40.395 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.550 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:40.398 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:27:57.562 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:40.409 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.496 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:50.344 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.502 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:50.350 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.515 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:28:50.362 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.550 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:50.397 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.550 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:50.354 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.551 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:50.355 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.552 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:50.400 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.563 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:28:50.367 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:07.563 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:28:50.411 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.476 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:00.323 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.521 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:00.369 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.534 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:00.382 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.569 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:00.373 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.570 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:00.374 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.571 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:00.419 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.573 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:00.421 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.584 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:00.388 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:17.594 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:00.442 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.556 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:10.403 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.562 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:10.410 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.578 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:10.426 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.614 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:10.418 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.617 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:10.421 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.630 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:10.434 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.635 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:10.483 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.638 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:10.486 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:27.651 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:10.499 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.287 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:11.134 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.288 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:11.136 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.296 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:11.144 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.300 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:11.105 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.305 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:11.109 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.318 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:11.122 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.332 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:11.180 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.333 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:11.181 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:28.341 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:11.189 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.910 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:14.757 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.912 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:14.760 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.926 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:14.774 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.960 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:14.764 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.963 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:14.768 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.967 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:14.815 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.968 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:14.817 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.977 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:14.781 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:31.982 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:14.830 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.571 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:20.418 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.575 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:20.423 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.586 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:20.434 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.623 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:20.427 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.625 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:20.429 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.635 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:20.439 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.641 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:20.489 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.645 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:20.493 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:37.655 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:20.503 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.599 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:30.447 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.604 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:30.452 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.618 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:30.466 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.654 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:30.501 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.655 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:30.503 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.656 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:30.460 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.658 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:30.462 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.668 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:30.516 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:47.672 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:30.476 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.235 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:31.083 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.236 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:31.084 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.243 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:31.048 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.245 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:31.049 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.248 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:31.096 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.254 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:31.058 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.265 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:31.113 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.266 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:31.114 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:48.278 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:31.126 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.618 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:40.465 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.620 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:40.469 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.631 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:40.480 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.670 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:40.474 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.670 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:40.518 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.671 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:40.475 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.672 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:40.520 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.682 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:40.487 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:28:57.683 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:40.531 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.768 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:50.616 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.770 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:50.618 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.786 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:29:50.634 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.818 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:50.666 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.819 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:50.623 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.820 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:50.668 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.823 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:50.627 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.833 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:29:50.681 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:07.836 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:29:50.641 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.678 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:00.526 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.680 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:00.528 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.694 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:00.542 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.729 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:00.576 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.729 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:00.533 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.730 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:00.578 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.730 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:00.534 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.741 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:00.589 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:17.742 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:00.546 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.709 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:10.557 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.712 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:10.561 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.723 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:10.572 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.763 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:10.611 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.763 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:10.567 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.764 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:10.612 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.767 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:10.572 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.777 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:10.625 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:27.781 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:10.586 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.744 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:20.592 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.790 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:20.595 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.792 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:20.597 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.805 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:20.610 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.816 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:20.664 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.830 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:20.678 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.880 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:20.728 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.885 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:20.733 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:37.898 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:20.746 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.733 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:30.581 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.780 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:30.628 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.782 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:30.630 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.783 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:30.588 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.785 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:30.633 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.790 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:30.595 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.802 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:30.650 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.804 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:30.652 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:47.807 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:30.612 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.753 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:40.601 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.798 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:40.647 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.801 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:40.605 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.802 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:40.606 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.813 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:40.661 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.818 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:40.622 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.850 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:40.698 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.851 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:40.700 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:29:57.863 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:40.711 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.824 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:50.673 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.826 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:50.674 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.840 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:30:50.688 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.873 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:50.677 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.874 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:50.679 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.886 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:30:50.691 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.889 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:50.738 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.892 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:50.740 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:07.904 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:30:50.752 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.800 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:00.648 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.846 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:00.694 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.857 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:00.706 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.895 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:00.699 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.896 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:00.700 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.908 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:00.712 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.912 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:00.760 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.913 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:00.761 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:17.926 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:00.774 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.865 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:10.712 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.868 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:10.716 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.879 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:10.728 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.917 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:10.765 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.918 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:10.722 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.919 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:10.723 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.919 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:10.767 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.930 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:10.735 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:27.934 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:10.782 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.843 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:20.691 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.891 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:20.739 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.893 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:20.741 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.895 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:20.700 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.896 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:20.744 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.898 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:20.702 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.905 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:20.754 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.908 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:20.756 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:37.909 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:20.713 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.906 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:30.754 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.910 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:30.758 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.921 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:30.770 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.958 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:30.806 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.958 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:30.762 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.959 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:30.763 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.959 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:30.807 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.970 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:30.775 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:30:47.971 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:30.819 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:02.959 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:45.806 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:02.962 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:45.811 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:02.977 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:45.825 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:03.015 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:45.863 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:03.016 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:45.820 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:03.017 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:45.822 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:03.018 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:45.866 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:03.032 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:45.837 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:03.035 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:45.883 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:07.967 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:50.815 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:07.971 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:50.819 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:07.981 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:31:50.830 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:08.019 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:50.867 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:08.020 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:50.869 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:08.023 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:50.828 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:08.025 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:50.830 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:08.031 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:31:50.879 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:08.036 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:31:50.841 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.004 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:05.853 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.007 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:05.855 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.021 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:05.869 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.055 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:05.904 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.057 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:05.905 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.060 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:05.865 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.061 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:05.866 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.071 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:05.919 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:23.075 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:05.880 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.018 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:10.866 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.021 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:10.870 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.035 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:10.884 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.070 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:10.874 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.071 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:10.876 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.073 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:10.921 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.074 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:10.922 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.081 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:10.886 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:28.084 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:10.933 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.064 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:25.912 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.066 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:25.914 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.080 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:25.928 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.115 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:25.963 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.117 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:25.965 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.118 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:25.923 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.120 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:25.924 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.131 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:25.980 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:43.134 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:25.939 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.073 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:30.921 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.076 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:30.925 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.088 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:30.937 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.120 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:30.924 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.120 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:30.969 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.121 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:30.926 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.123 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:30.972 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.134 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:30.938 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:48.136 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:30.984 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:55.155 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: Running main() from gmock_main.cc 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [----------] Global test environment set-up. 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (11440 ms) 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1809575 ms) 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1821015 ms total) 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [----------] Global test environment tear-down 2024-03-25T02:31:55.156 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1821015 ms total) 2024-03-25T02:31:55.157 INFO:tasks.workunit.client.0.smithi083.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-25T02:31:55.158 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.158 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40765 2024-03-25T02:31:55.158 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40765 2024-03-25T02:31:55.158 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.158 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41380 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41380 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41233 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41233 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40638 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40638 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40704 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40704 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41243 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41243 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.159 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40629 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40629 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40649 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40649 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41646 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41646 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40882 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40882 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41756 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41756 2024-03-25T02:31:55.160 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41028 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41028 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41465 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41465 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41802 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41802 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40931 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40931 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41286 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41286 2024-03-25T02:31:55.161 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.162 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41601 2024-03-25T02:31:55.162 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41601 2024-03-25T02:31:55.162 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:55.162 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41441 2024-03-25T02:31:55.162 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41441 2024-03-25T02:31:58.123 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:40.970 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.126 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:40.974 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.160 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:41.009 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.175 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:40.980 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.177 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:40.982 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.187 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:41.036 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.189 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:41.037 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.200 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:41.004 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.222 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:41.070 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:31:58.621 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: Running main() from gmock_main.cc 2024-03-25T02:31:58.621 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [----------] Global test environment set-up. 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6119 ms) 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (5560 ms) 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (1812818 ms) 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [----------] 3 tests from NeoRadosCmd (1824497 ms total) 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [----------] Global test environment tear-down 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [==========] 3 tests from 1 test suite ran. (1824497 ms total) 2024-03-25T02:31:58.622 INFO:tasks.workunit.client.0.smithi083.stdout: cmd: [ PASSED ] 3 tests. 2024-03-25T02:31:58.623 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41707 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41707 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41221 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41221 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40806 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40806 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41769 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41769 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41133 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41133 2024-03-25T02:31:58.624 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:31:58.625 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40622 2024-03-25T02:31:58.625 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40622 2024-03-25T02:32:01.959 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:44.807 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:01.972 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:44.821 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:01.977 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:44.782 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:01.984 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:44.789 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:01.991 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:44.839 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:01.994 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:44.842 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:02.067 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:44.916 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:02.072 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:44.877 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:02.076 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:44.924 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.151 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:51.000 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.158 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:51.006 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.183 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:32:51.031 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.207 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:51.012 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.216 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:51.020 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.226 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:51.074 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.228 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:51.077 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.232 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:32:51.037 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:08.244 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:32:51.092 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.236 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:01.084 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.291 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:01.139 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.313 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:01.162 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.338 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:01.186 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.339 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:01.188 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.345 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:01.150 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.347 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:01.152 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.369 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:01.174 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:18.370 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:01.218 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.195 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:06.043 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.200 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:06.049 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.218 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:06.067 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.257 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:06.062 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.259 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:06.064 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.269 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:06.117 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.272 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:06.120 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.278 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:06.083 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:23.287 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:06.135 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.266 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:11.114 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.270 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:11.119 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.272 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:11.077 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.281 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:11.086 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.300 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:11.104 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.300 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:11.149 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.341 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:11.190 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.351 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:11.200 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:28.378 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:11.226 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.283 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:16.131 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.284 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:16.133 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.300 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:16.149 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.330 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:16.178 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.333 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:16.182 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.343 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:16.148 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.346 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:16.151 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.348 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:16.197 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:33.364 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:16.168 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.342 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:31.190 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.347 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:31.195 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.379 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:31.228 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.399 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:31.203 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.402 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:31.207 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.405 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:31.253 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.407 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:31.256 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.430 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:31.235 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:48.441 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:31.290 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.143 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:31.947 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.144 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:31.949 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.155 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:31.960 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.244 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:32.093 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.245 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:32.094 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.259 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:32.108 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.260 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:32.108 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.261 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:32.110 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:49.271 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:32.120 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.371 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:36.220 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.376 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:36.225 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.395 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:36.244 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.422 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:36.227 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.423 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:36.272 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.427 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:36.232 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.428 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:36.277 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.441 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:36.246 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:32:53.444 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:36.292 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.396 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:46.245 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.398 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:46.247 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.422 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:46.270 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.448 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:46.297 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.449 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:46.254 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.451 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:46.256 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.451 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:46.299 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.474 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:46.323 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:03.475 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:46.280 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:06.388 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: Running main() from gmock_main.cc 2024-03-25T02:33:06.388 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] Global test environment set-up. 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6520 ms) 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (1804652 ms) 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3759 ms) 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3116 ms) 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-25T02:33:06.389 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3388 ms) 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3436 ms) 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3340 ms) 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3482 ms) 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3409 ms) 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3308 ms) 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3353 ms) 2024-03-25T02:33:06.390 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3783 ms) 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3327 ms) 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3278 ms) 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.Flush (3514 ms) 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3294 ms) 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3343 ms) 2024-03-25T02:33:06.391 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3499 ms) 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3179 ms) 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2683 ms) 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3612 ms) 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3287 ms) 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3471 ms) 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3788 ms) 2024-03-25T02:33:06.392 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-25T02:33:06.393 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3308 ms) 2024-03-25T02:33:06.393 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-25T02:33:06.393 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3531 ms) 2024-03-25T02:33:06.393 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 26 tests from LibRadosAio (1892660 ms total) 2024-03-25T02:33:06.393 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: 2024-03-25T02:33:06.393 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-25T02:33:08.414 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:51.262 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.418 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:51.267 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.436 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:33:51.284 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.465 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:51.270 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.466 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:51.271 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.468 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:51.316 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.472 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:51.320 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.479 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:33:51.284 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:08.488 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:33:51.337 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.443 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:01.292 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.447 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:01.296 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.471 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:01.320 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.498 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:01.303 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.500 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:01.304 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.500 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:01.349 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.503 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:01.352 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.521 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:01.326 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:18.525 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:01.374 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.533 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:11.382 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.586 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:11.435 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.592 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:11.397 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.597 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:11.401 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.613 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:11.462 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.622 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:11.427 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.647 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:11.495 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.651 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:11.500 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:28.674 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:11.522 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.509 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:16.355 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.510 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:16.359 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.524 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:16.372 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.555 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:16.360 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.558 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:16.407 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.559 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:16.364 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.563 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:16.412 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.573 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:16.378 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:33.578 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:16.427 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.740 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:31.588 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.743 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:31.592 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.788 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:31.636 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.788 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:31.593 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.791 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:31.596 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.808 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:31.656 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.811 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:31.660 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.820 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:31.625 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:33:48.841 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:31.690 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.861 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:43.710 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.865 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:43.714 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.873 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:43.722 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.875 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:43.680 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.922 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:43.727 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.936 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:43.784 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.949 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:43.754 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.967 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:43.816 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:00.979 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:43.828 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.641 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:46.489 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.642 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:46.491 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.653 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:46.502 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.688 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:46.537 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.690 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:46.495 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.691 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:46.496 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.691 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:46.540 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.701 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:46.506 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:03.702 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:46.551 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.781 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:51.630 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.786 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:51.634 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.805 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:34:51.654 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.836 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:51.684 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.838 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:51.687 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.846 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:51.651 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.854 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:51.659 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.856 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:34:51.705 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:08.870 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:34:51.675 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.821 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:01.670 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.825 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:01.674 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.856 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:01.705 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.885 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:01.734 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.886 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:01.691 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.887 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:01.692 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.890 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:01.739 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.911 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:01.716 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:18.913 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:01.762 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.760 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:11.608 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.810 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:11.614 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.813 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:11.618 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.824 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:11.673 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.833 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:11.638 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.848 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:11.696 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.874 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:11.723 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.875 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:11.724 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:28.896 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:11.745 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.870 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:16.718 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.871 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:16.676 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.873 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:16.722 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.880 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:16.685 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.887 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:16.736 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.897 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:16.702 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.931 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:16.780 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.933 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:16.782 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:34:33.952 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:16.801 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:03.988 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7069 ms) 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7811 ms) 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7259 ms) 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7648 ms) 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6811 ms) 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-25T02:35:03.989 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7814 ms) 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7697 ms) 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7508 ms) 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6983 ms) 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6767 ms) 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7665 ms) 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (8632 ms) 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-25T02:35:03.990 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7351 ms) 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6884 ms) 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6678 ms) 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7021 ms) 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (117598 ms total) 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [----------] Global test environment tear-down 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (2010258 ms total) 2024-03-25T02:35:03.991 INFO:tasks.workunit.client.0.smithi083.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40680 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40680 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41086 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41086 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=40724 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 40724 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41173 2024-03-25T02:35:03.993 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41173 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41186 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41186 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41544 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41544 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ for t in "${!pids[@]}" 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ pid=41825 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ wait 41825 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ exit 0 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ cleanup 2024-03-25T02:35:03.994 INFO:tasks.workunit.client.0.smithi083.stderr:+ pkill -P 40616 2024-03-25T02:35:04.101 INFO:tasks.workunit.client.0.smithi083.stderr:+ true 2024-03-25T02:35:04.102 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-25T02:35:04.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-25T02:35:04.134 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-03-25T02:35:04.134 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-25T02:35:04.475 DEBUG:teuthology.parallel:result is None 2024-03-25T02:35:04.476 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-25T02:35:04.502 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-25T02:35:04.502 DEBUG:teuthology.orchestra.run.smithi083:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-25T02:35:04.571 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-25T02:35:04.571 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T02:35:04.583 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-25T02:35:04.583 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.723+0000 7f3f70fa3640 10 monclient: get_monmap_and_config 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.723+0000 7f3f70fa3640 10 monclient: build_initial_monmap 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.723+0000 7f3f70fa3640 10 monclient: monmap: 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:04.724592+0000 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:04.724592+0000 2024-03-25T02:35:04.712 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:04.713 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:04.713 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:04.713 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:04.713 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:04.713 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:04.713 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:04.713 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.723+0000 7f3f70fa3640 1 Processor -- start 2024-03-25T02:35:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.723+0000 7f3f70fa3640 1 -- start start 2024-03-25T02:35:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.723+0000 7f3f70fa3640 10 monclient: init 2024-03-25T02:35:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f6c0af870 0x7f3f6c13e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f3f6c0af870 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c141030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f3f6c13e850 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f6c0b0a60 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6c0b0610 con 0x7f3f6c0b0a60 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f3f6c0ae820 con 0x7f3f6c0af870 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c141c80 con 0x7f3f6c13e850 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.724+0000 7f3f70fa3640 10 monclient(hunting): authenticate will time out at 3423.914490s 2024-03-25T02:35:04.716 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f69d9b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f6a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3f6c0af870 0x7f3f6c13e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f69d9b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c141030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50510/0 (socket says 172.21.15.83:50510) 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f69d9b640 1 -- 172.21.15.83:0/123902861 learned_addr learned my addr 172.21.15.83:0/123902861 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f69d9b640 10 monclient(hunting): get_auth_request con 0x7f3f6c13e850 auth_method 0 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f6a59c640 10 monclient(hunting): get_auth_request con 0x7f3f6c0af870 auth_method 0 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:04.717 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f6a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.725+0000 7f3f6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.726+0000 7f3f69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.726+0000 7f3f69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.726+0000 7f3f69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.726+0000 7f3f6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.726+0000 7f3f6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:04.718 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.726+0000 7f3f6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 10 monclient(hunting): handle_auth_done global_id 17204 payload 274 2024-03-25T02:35:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 1 -- 172.21.15.83:0/123902861 >> v2:172.21.15.153:3300/0 conn(0x7f3f6c0af870 msgr2=0x7f3f6c13e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 1 --2- 172.21.15.83:0/123902861 >> v2:172.21.15.153:3300/0 conn(0x7f3f6c0af870 0x7f3f6c13e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 1 -- 172.21.15.83:0/123902861 >> v1:172.21.15.153:6789/0 conn(0x7f3f6c0b0a60 legacy=0x7f3f6c141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 10 monclient: _finish_hunting 0 2024-03-25T02:35:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 1 monclient: found mon.noname-b 2024-03-25T02:35:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.727+0000 7f3f69d9b640 1 -- 172.21.15.83:0/123902861 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c0afc50 con 0x7f3f6c13e850 2024-03-25T02:35:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.729+0000 7f3f69d9b640 1 --2- 172.21.15.83:0/123902861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c141030 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009e50 tx=0x7f3f6002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04f5f3bf4afab53 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 1 -- 172.21.15.83:0/123902861 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6002c9e0 con 0x7f3f6c13e850 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: dump: 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:04.723 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: _finish_auth 0 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: _check_auth_tickets 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 1 -- 172.21.15.83:0/123902861 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f6002cb80 con 0x7f3f6c13e850 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 5 monclient: authenticate success, global_id 17204 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 1 -- 172.21.15.83:0/123902861 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f600343f0 con 0x7f3f6c13e850 2024-03-25T02:35:04.724 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 4 set_mon_vals no callback set 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f6959a640 10 monclient: dump: 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:04.725 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 10 monclient: shutdown 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 -- 172.21.15.83:0/123902861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 msgr2=0x7f3f6c141030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/123902861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c141030 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009e50 tx=0x7f3f6002be40 comp rx=0 tx=0).stop 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 -- 172.21.15.83:0/123902861 shutdown_connections 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/123902861 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c141030 unknown :-1 s=CLOSED pgs=3193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/123902861 >> v2:172.21.15.153:3300/0 conn(0x7f3f6c0af870 0x7f3f6c13e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 -- 172.21.15.83:0/123902861 >> 172.21.15.83:0/123902861 conn(0x7f3f6c0b4a10 msgr2=0x7f3f6c0b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 -- 172.21.15.83:0/123902861 shutdown_connections 2024-03-25T02:35:04.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.733+0000 7f3f70fa3640 1 -- 172.21.15.83:0/123902861 wait complete. 2024-03-25T02:35:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.734+0000 7f3f70fa3640 10 monclient: build_initial_monmap 2024-03-25T02:35:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.734+0000 7f3f70fa3640 10 monclient: monmap: 2024-03-25T02:35:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:04.730 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.734+0000 7f3f70fa3640 1 librados: starting msgr at 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.734+0000 7f3f70fa3640 1 librados: starting objecter 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 Processor -- start 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 -- start start 2024-03-25T02:35:04.731 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 librados: setting wanted keys 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 librados: calling monclient init 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient: init 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c0af870 0x7f3f6c13a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f3f6c0af870 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f6c0b0a60 0x7f3f6c13a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f3f6c0b0a60 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c1318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f3f6c13e850 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c06f7c0 con 0x7f3f6c0b0a60 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:04.732 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6c06f640 con 0x7f3f6c0af870 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6c06f940 con 0x7f3f6c13e850 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.735+0000 7f3f70fa3640 10 monclient(hunting): authenticate will time out at 3423.925665s 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6ad9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c1318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c0af870 0x7f3f6c13a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f69d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f6c0b0a60 0x7f3f6c13a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c0af870 0x7f3f6c13a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:59078/0 (socket says 172.21.15.83:59078) 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 1 -- 172.21.15.83:0/2971950358 learned_addr learned my addr 172.21.15.83:0/2971950358 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 10 monclient(hunting): get_auth_request con 0x7f3f6c0af870 auth_method 0 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:04.733 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6ad9d640 10 monclient(hunting): get_auth_request con 0x7f3f6c13e850 auth_method 0 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f69d9b640 10 monclient(hunting): get_auth_request con 0x7f3f6c0b0a60 auth_method 0 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f69d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.736+0000 7f3f6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.737+0000 7f3f6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.737+0000 7f3f6ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:04.734 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.737+0000 7f3f6ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.737+0000 7f3f6ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.737+0000 7f3f69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.737+0000 7f3f69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.737+0000 7f3f69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 10 monclient(hunting): handle_auth_done global_id 22012 payload 824 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 1 -- 172.21.15.83:0/2971950358 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 msgr2=0x7f3f6c1318b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c1318b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 1 -- 172.21.15.83:0/2971950358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f6c0b0a60 msgr2=0x7f3f6c13a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f6c0b0a60 0x7f3f6c13a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 10 monclient: _finish_hunting 0 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 1 monclient: found mon.noname-b 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:04.735 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:04.736 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.739+0000 7f3f6a59c640 1 -- 172.21.15.83:0/2971950358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6c1397d0 con 0x7f3f6c0af870 2024-03-25T02:35:04.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.742+0000 7f3f6a59c640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c0af870 0x7f3f6c13a030 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400e9e0 tx=0x7f3f6400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:04.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.742+0000 7f3f5affd640 10 client.?.objecter ms_handle_connect 0x7f3f6c0af870 2024-03-25T02:35:04.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.742+0000 7f3f5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:04.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 1 -- 172.21.15.83:0/2971950358 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64017070 con 0x7f3f6c0af870 2024-03-25T02:35:04.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:04.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:04.737 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: dump: 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: _finish_auth 0 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:04.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 1 -- 172.21.15.83:0/2971950358 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f6400c9d0 con 0x7f3f6c0af870 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 1 -- 172.21.15.83:0/2971950358 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400cdc0 con 0x7f3f6c0af870 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 5 monclient: authenticate success, global_id 22012 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f5affd640 10 monclient: dump: 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:04.739 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 10 monclient: _renew_subs 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6c070ad0 con 0x7f3f6c0af870 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f68d99640 4 set_mon_vals no callback set 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 10 client.22012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 10 monclient: _renew_subs 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f6c131df0 con 0x7f3f6c0af870 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.745+0000 7f3f70fa3640 1 librados: init done 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.746+0000 7f3f70fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:04.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.746+0000 7f3f70fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:04.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.746+0000 7f3f70fa3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:04.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.746+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f6c075810 con 0x7f3f6c0af870 2024-03-25T02:35:04.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.764+0000 7f3f5affd640 1 -- 172.21.15.83:0/2971950358 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f3f6401a070 con 0x7f3f6c0af870 2024-03-25T02:35:04.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.765+0000 7f3f5affd640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f4c06da70 0x7f3f4c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:04.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.765+0000 7f3f69d9b640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f4c06da70 0x7f3f4c06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:04.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.765+0000 7f3f69d9b640 10 monclient: get_auth_request con 0x7f3f4c06da70 auth_method 0 2024-03-25T02:35:04.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.766+0000 7f3f5affd640 10 client.22012.objecter ms_handle_connect 0x7f3f4c06da70 2024-03-25T02:35:04.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.766+0000 7f3f69d9b640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f4c06da70 0x7f3f4c06ff30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3f60035000 tx=0x7f3f60030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:04.757 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.768+0000 7f3f5affd640 1 -- 172.21.15.83:0/2971950358 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f3f64057ad0 con 0x7f3f6c0af870 2024-03-25T02:35:04.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.768+0000 7f3f5affd640 10 client.22012.objecter ms_dispatch 0x7f3f6c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:04.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.768+0000 7f3f5affd640 3 client.22012.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:04.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.768+0000 7f3f5affd640 3 client.22012.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:04.758 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:04.769+0000 7f3f5affd640 20 client.22012.objecter dump_active .. 0 homeless 2024-03-25T02:35:04.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.009+0000 7f3f5affd640 1 -- 172.21.15.83:0/2971950358 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3f64059a30 con 0x7f3f6c0af870 2024-03-25T02:35:04.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.010+0000 7f3f5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:04.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.010+0000 7f3f5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:05.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.216+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f6c076d20 con 0x7f3f4c06da70 2024-03-25T02:35:05.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.219+0000 7f3f5affd640 1 -- 172.21.15.83:0/2971950358 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116240 (secure 0 0 0) 0x7f3f6002c8e0 con 0x7f3f4c06da70 2024-03-25T02:35:05.210 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:35:05.210 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.221+0000 7f3f70fa3640 10 librados: watch_flush enter 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 10 librados: watch_flush exit 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 20 client.22012.objecter shutdown clearing up homeless session... 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 10 client.22012.objecter successfully canceled tick 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f4c06da70 msgr2=0x7f3f4c06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f4c06da70 0x7f3f4c06ff30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3f60035000 tx=0x7f3f60030000 comp rx=0 tx=0).stop 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 10 monclient: shutdown 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c0af870 msgr2=0x7f3f6c13a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.213 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c0af870 0x7f3f6c13a030 secure :-1 s=READY pgs=5526 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400e9e0 tx=0x7f3f6400eea0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 shutdown_connections 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3f4c06da70 0x7f3f4c06ff30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3f6c13e850 0x7f3f6c1318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3f6c0b0a60 0x7f3f6c13a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 --2- 172.21.15.83:0/2971950358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3f6c0af870 0x7f3f6c13a030 unknown :-1 s=CLOSED pgs=5526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 >> 172.21.15.83:0/2971950358 conn(0x7f3f6c0b4a10 msgr2=0x7f3f6c0b4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 shutdown_connections 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 -- 172.21.15.83:0/2971950358 wait complete. 2024-03-25T02:35:05.214 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.222+0000 7f3f70fa3640 1 librados: shutdown 2024-03-25T02:35:05.222 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":2299,"stamp":"2024-03-25T02:35:04.931421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":89,"num_write":121,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":559548,"kb_used_data":29636,"kb_used_omap":12,"kb_used_meta":529779,"kb_avail":754415172,"statfs":{"total":773094113280,"available":772521136128,"internally_reserved":0,"allocated":30347264,"data_stored":12899502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":542494256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":118,"apply_latency_ms":118,"commit_latency_ns":118000000,"apply_latency_ns":118000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000678"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2401,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.614411+0000","last_change":"2024-03-25T02:01:59.223551+0000","last_active":"2024-03-25T02:35:01.614411+0000","last_peered":"2024-03-25T02:35:01.614411+0000","last_clean":"2024-03-25T02:35:01.614411+0000","last_became_active":"2024-03-25T02:01:59.217800+0000","last_became_peered":"2024-03-25T02:01:59.217800+0000","last_unstale":"2024-03-25T02:35:01.614411+0000","last_undegraded":"2024-03-25T02:35:01.614411+0000","last_fullsized":"2024-03-25T02:35:01.614411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:52:12.956708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.291634+0000","last_change":"2024-03-25T02:01:45.802865+0000","last_active":"2024-03-25T02:34:59.291634+0000","last_peered":"2024-03-25T02:34:59.291634+0000","last_clean":"2024-03-25T02:34:59.291634+0000","last_became_active":"2024-03-25T02:01:45.737540+0000","last_became_peered":"2024-03-25T02:01:45.737540+0000","last_unstale":"2024-03-25T02:34:59.291634+0000","last_undegraded":"2024-03-25T02:34:59.291634+0000","last_fullsized":"2024-03-25T02:34:59.291634+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:53:36.729760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2410,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:02.601950+0000","last_change":"2024-03-25T02:01:57.033633+0000","last_active":"2024-03-25T02:35:02.601950+0000","last_peered":"2024-03-25T02:35:02.601950+0000","last_clean":"2024-03-25T02:35:02.601950+0000","last_became_active":"2024-03-25T02:01:57.014842+0000","last_became_peered":"2024-03-25T02:01:57.014842+0000","last_unstale":"2024-03-25T02:35:02.601950+0000","last_undegraded":"2024-03-25T02:35:02.601950+0000","last_fullsized":"2024-03-25T02:35:02.601950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:30:09.539808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.697841+0000","last_change":"2024-03-25T02:01:39.501653+0000","last_active":"2024-03-25T02:35:01.697841+0000","last_peered":"2024-03-25T02:35:01.697841+0000","last_clean":"2024-03-25T02:35:01.697841+0000","last_became_active":"2024-03-25T02:01:39.486234+0000","last_became_peered":"2024-03-25T02:01:39.486234+0000","last_unstale":"2024-03-25T02:35:01.697841+0000","last_undegraded":"2024-03-25T02:35:01.697841+0000","last_fullsized":"2024-03-25T02:35:01.697841+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:45:00.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.709178+0000","last_change":"2024-03-25T02:01:39.797384+0000","last_active":"2024-03-25T02:35:01.709178+0000","last_peered":"2024-03-25T02:35:01.709178+0000","last_clean":"2024-03-25T02:35:01.709178+0000","last_became_active":"2024-03-25T02:01:39.792053+0000","last_became_peered":"2024-03-25T02:01:39.792053+0000","last_unstale":"2024-03-25T02:35:01.709178+0000","last_undegraded":"2024-03-25T02:35:01.709178+0000","last_fullsized":"2024-03-25T02:35:01.709178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:38.937942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2409,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.365260+0000","last_change":"2024-03-25T02:01:42.883607+0000","last_active":"2024-03-25T02:35:03.365260+0000","last_peered":"2024-03-25T02:35:03.365260+0000","last_clean":"2024-03-25T02:35:03.365260+0000","last_became_active":"2024-03-25T02:01:42.877188+0000","last_became_peered":"2024-03-25T02:01:42.877188+0000","last_unstale":"2024-03-25T02:35:03.365260+0000","last_undegraded":"2024-03-25T02:35:03.365260+0000","last_fullsized":"2024-03-25T02:35:03.365260+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:55:25.547629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2409,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.298817+0000","last_change":"2024-03-25T02:01:45.794313+0000","last_active":"2024-03-25T02:34:59.298817+0000","last_peered":"2024-03-25T02:34:59.298817+0000","last_clean":"2024-03-25T02:34:59.298817+0000","last_became_active":"2024-03-25T02:01:45.785913+0000","last_became_peered":"2024-03-25T02:01:45.785913+0000","last_unstale":"2024-03-25T02:34:59.298817+0000","last_undegraded":"2024-03-25T02:34:59.298817+0000","last_fullsized":"2024-03-25T02:34:59.298817+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:46:04.871320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.302500+0000","last_change":"2024-03-25T02:01:47.960650+0000","last_active":"2024-03-25T02:34:59.302500+0000","last_peered":"2024-03-25T02:34:59.302500+0000","last_clean":"2024-03-25T02:34:59.302500+0000","last_became_active":"2024-03-25T02:01:47.939684+0000","last_became_peered":"2024-03-25T02:01:47.939684+0000","last_unstale":"2024-03-25T02:34:59.302500+0000","last_undegraded":"2024-03-25T02:34:59.302500+0000","last_fullsized":"2024-03-25T02:34:59.302500+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:51:58.276672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2424,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.342976+0000","last_change":"2024-03-25T02:01:58.313196+0000","last_active":"2024-03-25T02:35:03.342976+0000","last_peered":"2024-03-25T02:35:03.342976+0000","last_clean":"2024-03-25T02:35:03.342976+0000","last_became_active":"2024-03-25T02:01:58.306972+0000","last_became_peered":"2024-03-25T02:01:58.306972+0000","last_unstale":"2024-03-25T02:35:03.342976+0000","last_undegraded":"2024-03-25T02:35:03.342976+0000","last_fullsized":"2024-03-25T02:35:03.342976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:05:07.200253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.661493+0000","last_change":"2024-03-25T02:01:44.749489+0000","last_active":"2024-03-25T02:35:01.661493+0000","last_peered":"2024-03-25T02:35:01.661493+0000","last_clean":"2024-03-25T02:35:01.661493+0000","last_became_active":"2024-03-25T02:01:44.731330+0000","last_became_peered":"2024-03-25T02:01:44.731330+0000","last_unstale":"2024-03-25T02:35:01.661493+0000","last_undegraded":"2024-03-25T02:35:01.661493+0000","last_fullsized":"2024-03-25T02:35:01.661493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:40:48.117174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2437,"reported_epoch":1246,"state":"active+clean","last_fresh":"2024-03-25T02:35:00.425759+0000","last_change":"2024-03-25T02:01:38.014396+0000","last_active":"2024-03-25T02:35:00.425759+0000","last_peered":"2024-03-25T02:35:00.425759+0000","last_clean":"2024-03-25T02:35:00.425759+0000","last_became_active":"2024-03-25T02:01:38.008862+0000","last_became_peered":"2024-03-25T02:01:38.008862+0000","last_unstale":"2024-03-25T02:35:00.425759+0000","last_undegraded":"2024-03-25T02:35:00.425759+0000","last_fullsized":"2024-03-25T02:35:00.425759+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:48:26.588270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.624056+0000","last_change":"2024-03-25T02:01:55.459478+0000","last_active":"2024-03-25T02:35:01.624056+0000","last_peered":"2024-03-25T02:35:01.624056+0000","last_clean":"2024-03-25T02:35:01.624056+0000","last_became_active":"2024-03-25T02:01:55.452605+0000","last_became_peered":"2024-03-25T02:01:55.452605+0000","last_unstale":"2024-03-25T02:35:01.624056+0000","last_undegraded":"2024-03-25T02:35:01.624056+0000","last_fullsized":"2024-03-25T02:35:01.624056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:52:09.879541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"23'77","reported_seq":2503,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.347895+0000","last_change":"2024-03-25T02:00:26.482050+0000","last_active":"2024-03-25T02:34:58.347895+0000","last_peered":"2024-03-25T02:34:58.347895+0000","last_clean":"2024-03-25T02:34:58.347895+0000","last_became_active":"2024-03-25T02:00:25.840109+0000","last_became_peered":"2024-03-25T02:00:25.840109+0000","last_unstale":"2024-03-25T02:34:58.347895+0000","last_undegraded":"2024-03-25T02:34:58.347895+0000","last_fullsized":"2024-03-25T02:34:58.347895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_clean_scrub_stamp":"2024-03-25T02:00:15.236797+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:57:04.634251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.551661+0000","last_change":"2024-03-25T02:00:33.873073+0000","last_active":"2024-03-25T02:35:01.551661+0000","last_peered":"2024-03-25T02:35:01.551661+0000","last_clean":"2024-03-25T02:35:01.551661+0000","last_became_active":"2024-03-25T02:00:33.865870+0000","last_became_peered":"2024-03-25T02:00:33.865870+0000","last_unstale":"2024-03-25T02:35:01.551661+0000","last_undegraded":"2024-03-25T02:35:01.551661+0000","last_fullsized":"2024-03-25T02:35:01.551661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:52:25.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.625707+0000","last_change":"2024-03-25T02:01:42.911218+0000","last_active":"2024-03-25T02:35:01.625707+0000","last_peered":"2024-03-25T02:35:01.625707+0000","last_clean":"2024-03-25T02:35:01.625707+0000","last_became_active":"2024-03-25T02:01:42.904032+0000","last_became_peered":"2024-03-25T02:01:42.904032+0000","last_unstale":"2024-03-25T02:35:01.625707+0000","last_undegraded":"2024-03-25T02:35:01.625707+0000","last_fullsized":"2024-03-25T02:35:01.625707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:28:21.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.511421+0000","last_change":"2024-03-25T02:00:33.383977+0000","last_active":"2024-03-25T02:35:01.511421+0000","last_peered":"2024-03-25T02:35:01.511421+0000","last_clean":"2024-03-25T02:35:01.511421+0000","last_became_active":"2024-03-25T02:00:33.356691+0000","last_became_peered":"2024-03-25T02:00:33.356691+0000","last_unstale":"2024-03-25T02:35:01.511421+0000","last_undegraded":"2024-03-25T02:35:01.511421+0000","last_fullsized":"2024-03-25T02:35:01.511421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:57.213604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2406,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.700884+0000","last_change":"2024-03-25T02:01:39.128793+0000","last_active":"2024-03-25T02:35:01.700884+0000","last_peered":"2024-03-25T02:35:01.700884+0000","last_clean":"2024-03-25T02:35:01.700884+0000","last_became_active":"2024-03-25T02:01:39.122271+0000","last_became_peered":"2024-03-25T02:01:39.122271+0000","last_unstale":"2024-03-25T02:35:01.700884+0000","last_undegraded":"2024-03-25T02:35:01.700884+0000","last_fullsized":"2024-03-25T02:35:01.700884+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:57:32.356420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2419,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.344989+0000","last_change":"2024-03-25T02:00:33.459985+0000","last_active":"2024-03-25T02:35:03.344989+0000","last_peered":"2024-03-25T02:35:03.344989+0000","last_clean":"2024-03-25T02:35:03.344989+0000","last_became_active":"2024-03-25T02:00:33.429925+0000","last_became_peered":"2024-03-25T02:00:33.429925+0000","last_unstale":"2024-03-25T02:35:03.344989+0000","last_undegraded":"2024-03-25T02:35:03.344989+0000","last_fullsized":"2024-03-25T02:35:03.344989+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:09:49.004802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2398,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.356194+0000","last_change":"2024-03-25T02:01:53.314119+0000","last_active":"2024-03-25T02:34:58.356194+0000","last_peered":"2024-03-25T02:34:58.356194+0000","last_clean":"2024-03-25T02:34:58.356194+0000","last_became_active":"2024-03-25T02:01:53.306467+0000","last_became_peered":"2024-03-25T02:01:53.306467+0000","last_unstale":"2024-03-25T02:34:58.356194+0000","last_undegraded":"2024-03-25T02:34:58.356194+0000","last_fullsized":"2024-03-25T02:34:58.356194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:56:31.011637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.557938+0000","last_change":"2024-03-25T02:00:33.707296+0000","last_active":"2024-03-25T02:35:01.557938+0000","last_peered":"2024-03-25T02:35:01.557938+0000","last_clean":"2024-03-25T02:35:01.557938+0000","last_became_active":"2024-03-25T02:00:33.691900+0000","last_became_peered":"2024-03-25T02:00:33.691900+0000","last_unstale":"2024-03-25T02:35:01.557938+0000","last_undegraded":"2024-03-25T02:35:01.557938+0000","last_fullsized":"2024-03-25T02:35:01.557938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:26:57.139959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2406,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.676770+0000","last_change":"2024-03-25T02:01:39.897197+0000","last_active":"2024-03-25T02:35:01.676770+0000","last_peered":"2024-03-25T02:35:01.676770+0000","last_clean":"2024-03-25T02:35:01.676770+0000","last_became_active":"2024-03-25T02:01:39.890974+0000","last_became_peered":"2024-03-25T02:01:39.890974+0000","last_unstale":"2024-03-25T02:35:01.676770+0000","last_undegraded":"2024-03-25T02:35:01.676770+0000","last_fullsized":"2024-03-25T02:35:01.676770+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:25:57.375716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.706846+0000","last_change":"2024-03-25T02:00:33.702006+0000","last_active":"2024-03-25T02:35:01.706846+0000","last_peered":"2024-03-25T02:35:01.706846+0000","last_clean":"2024-03-25T02:35:01.706846+0000","last_became_active":"2024-03-25T02:00:33.693877+0000","last_became_peered":"2024-03-25T02:00:33.693877+0000","last_unstale":"2024-03-25T02:35:01.706846+0000","last_undegraded":"2024-03-25T02:35:01.706846+0000","last_fullsized":"2024-03-25T02:35:01.706846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:52:31.586659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.296971+0000","last_change":"2024-03-25T02:01:50.169634+0000","last_active":"2024-03-25T02:34:59.296971+0000","last_peered":"2024-03-25T02:34:59.296971+0000","last_clean":"2024-03-25T02:34:59.296971+0000","last_became_active":"2024-03-25T02:01:50.160922+0000","last_became_peered":"2024-03-25T02:01:50.160922+0000","last_unstale":"2024-03-25T02:34:59.296971+0000","last_undegraded":"2024-03-25T02:34:59.296971+0000","last_fullsized":"2024-03-25T02:34:59.296971+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:28:52.038897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.571339+0000","last_change":"2024-03-25T02:00:33.514196+0000","last_active":"2024-03-25T02:35:01.571339+0000","last_peered":"2024-03-25T02:35:01.571339+0000","last_clean":"2024-03-25T02:35:01.571339+0000","last_became_active":"2024-03-25T02:00:33.508976+0000","last_became_peered":"2024-03-25T02:00:33.508976+0000","last_unstale":"2024-03-25T02:35:01.571339+0000","last_undegraded":"2024-03-25T02:35:01.571339+0000","last_fullsized":"2024-03-25T02:35:01.571339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:14:11.807682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.300682+0000","last_change":"2024-03-25T02:01:59.814614+0000","last_active":"2024-03-25T02:34:59.300682+0000","last_peered":"2024-03-25T02:34:59.300682+0000","last_clean":"2024-03-25T02:34:59.300682+0000","last_became_active":"2024-03-25T02:01:59.807793+0000","last_became_peered":"2024-03-25T02:01:59.807793+0000","last_unstale":"2024-03-25T02:34:59.300682+0000","last_undegraded":"2024-03-25T02:34:59.300682+0000","last_fullsized":"2024-03-25T02:34:59.300682+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:42:56.823793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2421,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:02.589473+0000","last_change":"2024-03-25T02:00:33.937499+0000","last_active":"2024-03-25T02:35:02.589473+0000","last_peered":"2024-03-25T02:35:02.589473+0000","last_clean":"2024-03-25T02:35:02.589473+0000","last_became_active":"2024-03-25T02:00:33.909846+0000","last_became_peered":"2024-03-25T02:00:33.909846+0000","last_unstale":"2024-03-25T02:35:02.589473+0000","last_undegraded":"2024-03-25T02:35:02.589473+0000","last_fullsized":"2024-03-25T02:35:02.589473+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:26:26.798933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.569217+0000","last_change":"2024-03-25T02:01:37.864558+0000","last_active":"2024-03-25T02:35:01.569217+0000","last_peered":"2024-03-25T02:35:01.569217+0000","last_clean":"2024-03-25T02:35:01.569217+0000","last_became_active":"2024-03-25T02:01:37.856709+0000","last_became_peered":"2024-03-25T02:01:37.856709+0000","last_unstale":"2024-03-25T02:35:01.569217+0000","last_undegraded":"2024-03-25T02:35:01.569217+0000","last_fullsized":"2024-03-25T02:35:01.569217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:25:19.300671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2410,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.581481+0000","last_change":"2024-03-25T02:00:33.801095+0000","last_active":"2024-03-25T02:35:01.581481+0000","last_peered":"2024-03-25T02:35:01.581481+0000","last_clean":"2024-03-25T02:35:01.581481+0000","last_became_active":"2024-03-25T02:00:33.792626+0000","last_became_peered":"2024-03-25T02:00:33.792626+0000","last_unstale":"2024-03-25T02:35:01.581481+0000","last_undegraded":"2024-03-25T02:35:01.581481+0000","last_fullsized":"2024-03-25T02:35:01.581481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:02:09.659440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2408,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.363375+0000","last_change":"2024-03-25T02:01:54.523614+0000","last_active":"2024-03-25T02:35:03.363375+0000","last_peered":"2024-03-25T02:35:03.363375+0000","last_clean":"2024-03-25T02:35:03.363375+0000","last_became_active":"2024-03-25T02:01:54.518327+0000","last_became_peered":"2024-03-25T02:01:54.518327+0000","last_unstale":"2024-03-25T02:35:03.363375+0000","last_undegraded":"2024-03-25T02:35:03.363375+0000","last_fullsized":"2024-03-25T02:35:03.363375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:28:31.375645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.643728+0000","last_change":"2024-03-25T02:01:53.995287+0000","last_active":"2024-03-25T02:35:01.643728+0000","last_peered":"2024-03-25T02:35:01.643728+0000","last_clean":"2024-03-25T02:35:01.643728+0000","last_became_active":"2024-03-25T02:01:53.989515+0000","last_became_peered":"2024-03-25T02:01:53.989515+0000","last_unstale":"2024-03-25T02:35:01.643728+0000","last_undegraded":"2024-03-25T02:35:01.643728+0000","last_fullsized":"2024-03-25T02:35:01.643728+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:43:06.616302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.647118+0000","last_change":"2024-03-25T02:01:55.429116+0000","last_active":"2024-03-25T02:35:01.647118+0000","last_peered":"2024-03-25T02:35:01.647118+0000","last_clean":"2024-03-25T02:35:01.647118+0000","last_became_active":"2024-03-25T02:01:55.419583+0000","last_became_peered":"2024-03-25T02:01:55.419583+0000","last_unstale":"2024-03-25T02:35:01.647118+0000","last_undegraded":"2024-03-25T02:35:01.647118+0000","last_fullsized":"2024-03-25T02:35:01.647118+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:54:42.345480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.704922+0000","last_change":"2024-03-25T02:01:56.827003+0000","last_active":"2024-03-25T02:35:01.704922+0000","last_peered":"2024-03-25T02:35:01.704922+0000","last_clean":"2024-03-25T02:35:01.704922+0000","last_became_active":"2024-03-25T02:01:56.799197+0000","last_became_peered":"2024-03-25T02:01:56.799197+0000","last_unstale":"2024-03-25T02:35:01.704922+0000","last_undegraded":"2024-03-25T02:35:01.704922+0000","last_fullsized":"2024-03-25T02:35:01.704922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:49:03.924799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2401,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.632313+0000","last_change":"2024-03-25T02:01:50.245600+0000","last_active":"2024-03-25T02:35:01.632313+0000","last_peered":"2024-03-25T02:35:01.632313+0000","last_clean":"2024-03-25T02:35:01.632313+0000","last_became_active":"2024-03-25T02:01:50.226409+0000","last_became_peered":"2024-03-25T02:01:50.226409+0000","last_unstale":"2024-03-25T02:35:01.632313+0000","last_undegraded":"2024-03-25T02:35:01.632313+0000","last_fullsized":"2024-03-25T02:35:01.632313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:33:00.764629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2397,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.354373+0000","last_change":"2024-03-25T02:01:39.447936+0000","last_active":"2024-03-25T02:34:58.354373+0000","last_peered":"2024-03-25T02:34:58.354373+0000","last_clean":"2024-03-25T02:34:58.354373+0000","last_became_active":"2024-03-25T02:01:39.437348+0000","last_became_peered":"2024-03-25T02:01:39.437348+0000","last_unstale":"2024-03-25T02:34:58.354373+0000","last_undegraded":"2024-03-25T02:34:58.354373+0000","last_fullsized":"2024-03-25T02:34:58.354373+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:31:57.241440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.285476+0000","last_change":"2024-03-25T02:01:43.296920+0000","last_active":"2024-03-25T02:34:59.285476+0000","last_peered":"2024-03-25T02:34:59.285476+0000","last_clean":"2024-03-25T02:34:59.285476+0000","last_became_active":"2024-03-25T02:01:43.285633+0000","last_became_peered":"2024-03-25T02:01:43.285633+0000","last_unstale":"2024-03-25T02:34:59.285476+0000","last_undegraded":"2024-03-25T02:34:59.285476+0000","last_fullsized":"2024-03-25T02:34:59.285476+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:17:06.380180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2397,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.345296+0000","last_change":"2024-03-25T02:01:41.663556+0000","last_active":"2024-03-25T02:34:58.345296+0000","last_peered":"2024-03-25T02:34:58.345296+0000","last_clean":"2024-03-25T02:34:58.345296+0000","last_became_active":"2024-03-25T02:01:41.631996+0000","last_became_peered":"2024-03-25T02:01:41.631996+0000","last_unstale":"2024-03-25T02:34:58.345296+0000","last_undegraded":"2024-03-25T02:34:58.345296+0000","last_fullsized":"2024-03-25T02:34:58.345296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:56:07.666478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.545302+0000","last_change":"2024-03-25T02:01:44.130872+0000","last_active":"2024-03-25T02:35:01.545302+0000","last_peered":"2024-03-25T02:35:01.545302+0000","last_clean":"2024-03-25T02:35:01.545302+0000","last_became_active":"2024-03-25T02:01:44.123357+0000","last_became_peered":"2024-03-25T02:01:44.123357+0000","last_unstale":"2024-03-25T02:35:01.545302+0000","last_undegraded":"2024-03-25T02:35:01.545302+0000","last_fullsized":"2024-03-25T02:35:01.545302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:47:38.979247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2434,"reported_epoch":1246,"state":"active+clean","last_fresh":"2024-03-25T02:35:00.433598+0000","last_change":"2024-03-25T02:01:40.283761+0000","last_active":"2024-03-25T02:35:00.433598+0000","last_peered":"2024-03-25T02:35:00.433598+0000","last_clean":"2024-03-25T02:35:00.433598+0000","last_became_active":"2024-03-25T02:01:40.275307+0000","last_became_peered":"2024-03-25T02:01:40.275307+0000","last_unstale":"2024-03-25T02:35:00.433598+0000","last_undegraded":"2024-03-25T02:35:00.433598+0000","last_fullsized":"2024-03-25T02:35:00.433598+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:28.413233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.295133+0000","last_change":"2024-03-25T02:01:50.267025+0000","last_active":"2024-03-25T02:34:59.295133+0000","last_peered":"2024-03-25T02:34:59.295133+0000","last_clean":"2024-03-25T02:34:59.295133+0000","last_became_active":"2024-03-25T02:01:50.240201+0000","last_became_peered":"2024-03-25T02:01:50.240201+0000","last_unstale":"2024-03-25T02:34:59.295133+0000","last_undegraded":"2024-03-25T02:34:59.295133+0000","last_fullsized":"2024-03-25T02:34:59.295133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:56.028592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.703082+0000","last_change":"2024-03-25T02:01:56.819977+0000","last_active":"2024-03-25T02:35:01.703082+0000","last_peered":"2024-03-25T02:35:01.703082+0000","last_clean":"2024-03-25T02:35:01.703082+0000","last_became_active":"2024-03-25T02:01:56.787332+0000","last_became_peered":"2024-03-25T02:01:56.787332+0000","last_unstale":"2024-03-25T02:35:01.703082+0000","last_undegraded":"2024-03-25T02:35:01.703082+0000","last_fullsized":"2024-03-25T02:35:01.703082+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:10:10.439975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2402,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.565652+0000","last_change":"2024-03-25T02:01:40.340111+0000","last_active":"2024-03-25T02:35:01.565652+0000","last_peered":"2024-03-25T02:35:01.565652+0000","last_clean":"2024-03-25T02:35:01.565652+0000","last_became_active":"2024-03-25T02:01:40.334290+0000","last_became_peered":"2024-03-25T02:01:40.334290+0000","last_unstale":"2024-03-25T02:35:01.565652+0000","last_undegraded":"2024-03-25T02:35:01.565652+0000","last_fullsized":"2024-03-25T02:35:01.565652+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:51.615630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":20,"seq":85899346334,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70872,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300968,"statfs":{"total":96636764160,"available":96564191232,"internally_reserved":0,"allocated":3674112,"data_stored":1502020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346340,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71248,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300592,"statfs":{"total":96636764160,"available":96563806208,"internally_reserved":0,"allocated":4063232,"data_stored":1891806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379042,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70900,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300940,"statfs":{"total":96636764160,"available":96564162560,"internally_reserved":0,"allocated":3702784,"data_stored":1517254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379044,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66736,"kb_used_data":3612,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305104,"statfs":{"total":96636764160,"available":96568426496,"internally_reserved":0,"allocated":3698688,"data_stored":1517249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71276,"kb_used_data":4004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300564,"statfs":{"total":96636764160,"available":96563777536,"internally_reserved":0,"allocated":4100096,"data_stored":1915089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66736,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305104,"statfs":{"total":96636764160,"available":96568426496,"internally_reserved":0,"allocated":3702784,"data_stored":1521345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70872,"kb_used_data":3600,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300968,"statfs":{"total":96636764160,"available":96564191232,"internally_reserved":0,"allocated":3686400,"data_stored":1509825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542564,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70908,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300932,"statfs":{"total":96636764160,"available":96564154368,"internally_reserved":0,"allocated":3719168,"data_stored":1524914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-25T02:35:05.224 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-25T02:35:05.325 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.336+0000 7fab67faf640 10 monclient: get_monmap_and_config 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.336+0000 7fab67faf640 10 monclient: build_initial_monmap 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.336+0000 7fab67faf640 10 monclient: monmap: 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:05.338222+0000 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:05.338222+0000 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:05.326 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:05.327 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.337+0000 7fab67faf640 1 Processor -- start 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.337+0000 7fab67faf640 1 -- start start 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.337+0000 7fab67faf640 10 monclient: init 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600a9b70 0x7fab601437c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient(hunting): picked mon.noname-b con 0x7fab600a9b70 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab600b33f0 0x7fab60140320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient(hunting): picked mon.noname-a con 0x7fab600b33f0 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:05.329 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab600a9f50 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab600abae0 con 0x7fab600a9f50 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fab600abdc0 con 0x7fab600b33f0 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fab600ac0a0 con 0x7fab600a9b70 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab67faf640 10 monclient(hunting): authenticate will time out at 3424.528190s 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab65d4b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600a9b70 0x7fab601437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.338+0000 7fab6554a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fab600b33f0 0x7fab60140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6654c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fab600a9f50 0x7fab60146680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33078/0 (socket says 172.21.15.83:33078) 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6654c640 1 -- 172.21.15.83:0/3816074179 learned_addr learned my addr 172.21.15.83:0/3816074179 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6554a640 10 monclient(hunting): get_auth_request con 0x7fab600b33f0 auth_method 0 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6554a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6554a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6554a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab65d4b640 10 monclient(hunting): get_auth_request con 0x7fab600a9b70 auth_method 0 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab65d4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab65d4b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab65d4b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6554a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6554a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab6554a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab65d4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab65d4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.339+0000 7fab65d4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab64d49640 1 -- 172.21.15.83:0/3816074179 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221100292 0 0) 0x7fab600abae0 con 0x7fab600a9f50 2024-03-25T02:35:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab64d49640 10 monclient(hunting): _negotiate 2024-03-25T02:35:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab64d49640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab64d49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab64d49640 10 monclient(hunting): my global_id is 22015 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab64d49640 1 -- 172.21.15.83:0/3816074179 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab4c003520 con 0x7fab600a9f50 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 10 monclient(hunting): handle_auth_done global_id 17210 payload 274 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 1 -- 172.21.15.83:0/3816074179 >> v2:172.21.15.83:3300/0 conn(0x7fab600b33f0 msgr2=0x7fab60140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 1 --2- 172.21.15.83:0/3816074179 >> v2:172.21.15.83:3300/0 conn(0x7fab600b33f0 0x7fab60140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 1 -- 172.21.15.83:0/3816074179 >> v1:172.21.15.153:6789/0 conn(0x7fab600a9f50 legacy=0x7fab60146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 10 monclient: _finish_hunting 0 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 1 monclient: found mon.noname-b 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.341+0000 7fab65d4b640 1 -- 172.21.15.83:0/3816074179 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab601432a0 con 0x7fab600a9b70 2024-03-25T02:35:05.333 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.343+0000 7fab65d4b640 1 --2- 172.21.15.83:0/3816074179 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600a9b70 0x7fab601437c0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7fab500097a0 tx=0x7fab5002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed10e5ec2cae0da server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 1 -- 172.21.15.83:0/3816074179 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50035020 con 0x7fab600a9b70 2024-03-25T02:35:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:05.336 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: dump: 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: _finish_auth 0 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: _check_auth_tickets 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:05.337 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 1 -- 172.21.15.83:0/3816074179 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab50031830 con 0x7fab600a9b70 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 1 -- 172.21.15.83:0/3816074179 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50031c00 con 0x7fab600a9b70 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab64d49640 10 monclient: dump: 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.338 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.339 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab67faf640 5 monclient: authenticate success, global_id 17210 2024-03-25T02:35:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab67faf640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:05.339 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.347+0000 7fab67faf640 4 set_mon_vals no callback set 2024-03-25T02:35:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 10 monclient: shutdown 2024-03-25T02:35:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 -- 172.21.15.83:0/3816074179 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600a9b70 msgr2=0x7fab601437c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 --2- 172.21.15.83:0/3816074179 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600a9b70 0x7fab601437c0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7fab500097a0 tx=0x7fab5002b8d0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.342 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 -- 172.21.15.83:0/3816074179 shutdown_connections 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 --2- 172.21.15.83:0/3816074179 >> v2:172.21.15.83:3300/0 conn(0x7fab600b33f0 0x7fab60140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 --2- 172.21.15.83:0/3816074179 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600a9b70 0x7fab601437c0 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 -- 172.21.15.83:0/3816074179 >> 172.21.15.83:0/3816074179 conn(0x7fab60067e30 msgr2=0x7fab600b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 -- 172.21.15.83:0/3816074179 shutdown_connections 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 1 -- 172.21.15.83:0/3816074179 wait complete. 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 10 monclient: build_initial_monmap 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.348+0000 7fab67faf640 10 monclient: monmap: 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.343 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 librados: starting msgr at 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 librados: starting objecter 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 Processor -- start 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 -- start start 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 librados: setting wanted keys 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 librados: calling monclient init 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient: init 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:35:05.344 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab600a9b70 0x7fab600b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): picked mon.noname-b con 0x7fab600a9b70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab600a9f50 0x7fab600b7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): picked mon.noname-a con 0x7fab600a9f50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600b33f0 0x7fab600b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): picked mon.noname-c con 0x7fab600b33f0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fab60140860 con 0x7fab600a9f50 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fab60143f10 con 0x7fab600a9b70 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fab601409e0 con 0x7fab600b33f0 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.349+0000 7fab67faf640 10 monclient(hunting): authenticate will time out at 3424.539907s 2024-03-25T02:35:05.345 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6554a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab600a9f50 0x7fab600b7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6654c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600b33f0 0x7fab600b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab65d4b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab600a9b70 0x7fab600b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6654c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600b33f0 0x7fab600b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50548/0 (socket says 172.21.15.83:50548) 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6654c640 1 -- 172.21.15.83:0/3218217550 learned_addr learned my addr 172.21.15.83:0/3218217550 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6654c640 10 monclient(hunting): get_auth_request con 0x7fab600b33f0 auth_method 0 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6654c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6654c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6654c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6554a640 10 monclient(hunting): get_auth_request con 0x7fab600a9f50 auth_method 0 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6554a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6554a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.350+0000 7fab6554a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab65d4b640 10 monclient(hunting): get_auth_request con 0x7fab600a9b70 auth_method 0 2024-03-25T02:35:05.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab65d4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab65d4b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab65d4b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab6654c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab6654c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab6654c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab6554a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab6554a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab6554a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab65d4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab65d4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.347 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.351+0000 7fab65d4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 10 monclient(hunting): handle_auth_done global_id 15342 payload 824 2024-03-25T02:35:05.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 1 -- 172.21.15.83:0/3218217550 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600b33f0 msgr2=0x7fab600b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600b33f0 0x7fab600b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 1 -- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab600a9b70 msgr2=0x7fab600b7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab600a9b70 0x7fab600b7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 10 monclient: _finish_hunting 0 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 1 monclient: found mon.noname-a 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.354+0000 7fab6554a640 1 -- 172.21.15.83:0/3218217550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab600b9630 con 0x7fab600a9f50 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.356+0000 7fab6554a640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab600a9f50 0x7fab600b7240 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00e840 tx=0x7fab5c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.356+0000 7fab567fc640 10 client.?.objecter ms_handle_connect 0x7fab600a9f50 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.356+0000 7fab567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 1 -- 172.21.15.83:0/3218217550 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c017070 con 0x7fab600a9f50 2024-03-25T02:35:05.350 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: dump: 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: _finish_auth 0 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: _check_auth_tickets 2024-03-25T02:35:05.351 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 1 -- 172.21.15.83:0/3218217550 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab5c00c730 con 0x7fab600a9f50 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 5 monclient: authenticate success, global_id 15342 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 10 monclient: _renew_subs 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 1 -- 172.21.15.83:0/3218217550 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5c00cb00 con 0x7fab600a9f50 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab600b7780 con 0x7fab600a9f50 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab57fff640 4 set_mon_vals no callback set 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab567fc640 10 monclient: dump: 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.352 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 10 client.15342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 10 monclient: _renew_subs 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab600b1e30 con 0x7fab600a9f50 2024-03-25T02:35:05.353 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.360+0000 7fab67faf640 1 librados: init done 2024-03-25T02:35:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.361+0000 7fab67faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.361+0000 7fab67faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.361+0000 7fab67faf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:05.354 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.361+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab600b8420 con 0x7fab600a9f50 2024-03-25T02:35:05.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.608+0000 7fab567fc640 1 -- 172.21.15.83:0/3218217550 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fab5c01a070 con 0x7fab600a9f50 2024-03-25T02:35:05.598 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.609+0000 7fab567fc640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fab4406da70 0x7fab4406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.609+0000 7fab65d4b640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fab4406da70 0x7fab4406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.609+0000 7fab65d4b640 10 monclient: get_auth_request con 0x7fab4406da70 auth_method 0 2024-03-25T02:35:05.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.610+0000 7fab65d4b640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fab4406da70 0x7fab4406ff30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fab50036000 tx=0x7fab50008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:05.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.610+0000 7fab567fc640 10 client.15342.objecter ms_handle_connect 0x7fab4406da70 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.622+0000 7fab567fc640 1 -- 172.21.15.83:0/3218217550 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7fab5c0578e0 con 0x7fab600a9f50 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.622+0000 7fab567fc640 10 client.15342.objecter ms_dispatch 0x7fab60000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.622+0000 7fab567fc640 3 client.15342.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.622+0000 7fab567fc640 3 client.15342.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.622+0000 7fab567fc640 20 client.15342.objecter dump_active .. 0 homeless 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.622+0000 7fab567fc640 1 -- 172.21.15.83:0/3218217550 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fab5c059840 con 0x7fab600a9f50 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.623+0000 7fab567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:05.612 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.623+0000 7fab567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:05.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.794+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab600b9470 con 0x7fab4406da70 2024-03-25T02:35:05.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.798+0000 7fab567fc640 1 -- 172.21.15.83:0/3218217550 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116240 (secure 0 0 0) 0x7fab50046440 con 0x7fab4406da70 2024-03-25T02:35:05.788 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:35:05.789 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 10 librados: watch_flush enter 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 10 librados: watch_flush exit 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 20 client.15342.objecter shutdown clearing up homeless session... 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 10 client.15342.objecter successfully canceled tick 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fab4406da70 msgr2=0x7fab4406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fab4406da70 0x7fab4406ff30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fab50036000 tx=0x7fab50008030 comp rx=0 tx=0).stop 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 10 monclient: shutdown 2024-03-25T02:35:05.791 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab600a9f50 msgr2=0x7fab600b7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab600a9f50 0x7fab600b7240 secure :-1 s=READY pgs=3193 cs=0 l=1 rev1=1 crypto rx=0x7fab5c00e840 tx=0x7fab5c00ed00 comp rx=0 tx=0).stop 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 shutdown_connections 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fab4406da70 0x7fab4406ff30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fab600b33f0 0x7fab600b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fab600a9f50 0x7fab600b7240 unknown :-1 s=CLOSED pgs=3193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 --2- 172.21.15.83:0/3218217550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab600a9b70 0x7fab600b7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 >> 172.21.15.83:0/3218217550 conn(0x7fab60067e30 msgr2=0x7fab6013d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 shutdown_connections 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 -- 172.21.15.83:0/3218217550 wait complete. 2024-03-25T02:35:05.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.800+0000 7fab67faf640 1 librados: shutdown 2024-03-25T02:35:05.802 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":2299,"stamp":"2024-03-25T02:35:04.931421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":89,"num_write":121,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":559548,"kb_used_data":29636,"kb_used_omap":12,"kb_used_meta":529779,"kb_avail":754415172,"statfs":{"total":773094113280,"available":772521136128,"internally_reserved":0,"allocated":30347264,"data_stored":12899502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":542494256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":118,"apply_latency_ms":118,"commit_latency_ns":118000000,"apply_latency_ns":118000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000678"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2401,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.614411+0000","last_change":"2024-03-25T02:01:59.223551+0000","last_active":"2024-03-25T02:35:01.614411+0000","last_peered":"2024-03-25T02:35:01.614411+0000","last_clean":"2024-03-25T02:35:01.614411+0000","last_became_active":"2024-03-25T02:01:59.217800+0000","last_became_peered":"2024-03-25T02:01:59.217800+0000","last_unstale":"2024-03-25T02:35:01.614411+0000","last_undegraded":"2024-03-25T02:35:01.614411+0000","last_fullsized":"2024-03-25T02:35:01.614411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:52:12.956708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.291634+0000","last_change":"2024-03-25T02:01:45.802865+0000","last_active":"2024-03-25T02:34:59.291634+0000","last_peered":"2024-03-25T02:34:59.291634+0000","last_clean":"2024-03-25T02:34:59.291634+0000","last_became_active":"2024-03-25T02:01:45.737540+0000","last_became_peered":"2024-03-25T02:01:45.737540+0000","last_unstale":"2024-03-25T02:34:59.291634+0000","last_undegraded":"2024-03-25T02:34:59.291634+0000","last_fullsized":"2024-03-25T02:34:59.291634+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:53:36.729760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2410,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:02.601950+0000","last_change":"2024-03-25T02:01:57.033633+0000","last_active":"2024-03-25T02:35:02.601950+0000","last_peered":"2024-03-25T02:35:02.601950+0000","last_clean":"2024-03-25T02:35:02.601950+0000","last_became_active":"2024-03-25T02:01:57.014842+0000","last_became_peered":"2024-03-25T02:01:57.014842+0000","last_unstale":"2024-03-25T02:35:02.601950+0000","last_undegraded":"2024-03-25T02:35:02.601950+0000","last_fullsized":"2024-03-25T02:35:02.601950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:30:09.539808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.697841+0000","last_change":"2024-03-25T02:01:39.501653+0000","last_active":"2024-03-25T02:35:01.697841+0000","last_peered":"2024-03-25T02:35:01.697841+0000","last_clean":"2024-03-25T02:35:01.697841+0000","last_became_active":"2024-03-25T02:01:39.486234+0000","last_became_peered":"2024-03-25T02:01:39.486234+0000","last_unstale":"2024-03-25T02:35:01.697841+0000","last_undegraded":"2024-03-25T02:35:01.697841+0000","last_fullsized":"2024-03-25T02:35:01.697841+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:45:00.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.709178+0000","last_change":"2024-03-25T02:01:39.797384+0000","last_active":"2024-03-25T02:35:01.709178+0000","last_peered":"2024-03-25T02:35:01.709178+0000","last_clean":"2024-03-25T02:35:01.709178+0000","last_became_active":"2024-03-25T02:01:39.792053+0000","last_became_peered":"2024-03-25T02:01:39.792053+0000","last_unstale":"2024-03-25T02:35:01.709178+0000","last_undegraded":"2024-03-25T02:35:01.709178+0000","last_fullsized":"2024-03-25T02:35:01.709178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:38.937942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2409,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.365260+0000","last_change":"2024-03-25T02:01:42.883607+0000","last_active":"2024-03-25T02:35:03.365260+0000","last_peered":"2024-03-25T02:35:03.365260+0000","last_clean":"2024-03-25T02:35:03.365260+0000","last_became_active":"2024-03-25T02:01:42.877188+0000","last_became_peered":"2024-03-25T02:01:42.877188+0000","last_unstale":"2024-03-25T02:35:03.365260+0000","last_undegraded":"2024-03-25T02:35:03.365260+0000","last_fullsized":"2024-03-25T02:35:03.365260+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:55:25.547629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2409,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.298817+0000","last_change":"2024-03-25T02:01:45.794313+0000","last_active":"2024-03-25T02:34:59.298817+0000","last_peered":"2024-03-25T02:34:59.298817+0000","last_clean":"2024-03-25T02:34:59.298817+0000","last_became_active":"2024-03-25T02:01:45.785913+0000","last_became_peered":"2024-03-25T02:01:45.785913+0000","last_unstale":"2024-03-25T02:34:59.298817+0000","last_undegraded":"2024-03-25T02:34:59.298817+0000","last_fullsized":"2024-03-25T02:34:59.298817+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:46:04.871320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.302500+0000","last_change":"2024-03-25T02:01:47.960650+0000","last_active":"2024-03-25T02:34:59.302500+0000","last_peered":"2024-03-25T02:34:59.302500+0000","last_clean":"2024-03-25T02:34:59.302500+0000","last_became_active":"2024-03-25T02:01:47.939684+0000","last_became_peered":"2024-03-25T02:01:47.939684+0000","last_unstale":"2024-03-25T02:34:59.302500+0000","last_undegraded":"2024-03-25T02:34:59.302500+0000","last_fullsized":"2024-03-25T02:34:59.302500+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:51:58.276672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2424,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.342976+0000","last_change":"2024-03-25T02:01:58.313196+0000","last_active":"2024-03-25T02:35:03.342976+0000","last_peered":"2024-03-25T02:35:03.342976+0000","last_clean":"2024-03-25T02:35:03.342976+0000","last_became_active":"2024-03-25T02:01:58.306972+0000","last_became_peered":"2024-03-25T02:01:58.306972+0000","last_unstale":"2024-03-25T02:35:03.342976+0000","last_undegraded":"2024-03-25T02:35:03.342976+0000","last_fullsized":"2024-03-25T02:35:03.342976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:05:07.200253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.661493+0000","last_change":"2024-03-25T02:01:44.749489+0000","last_active":"2024-03-25T02:35:01.661493+0000","last_peered":"2024-03-25T02:35:01.661493+0000","last_clean":"2024-03-25T02:35:01.661493+0000","last_became_active":"2024-03-25T02:01:44.731330+0000","last_became_peered":"2024-03-25T02:01:44.731330+0000","last_unstale":"2024-03-25T02:35:01.661493+0000","last_undegraded":"2024-03-25T02:35:01.661493+0000","last_fullsized":"2024-03-25T02:35:01.661493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:40:48.117174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2437,"reported_epoch":1246,"state":"active+clean","last_fresh":"2024-03-25T02:35:00.425759+0000","last_change":"2024-03-25T02:01:38.014396+0000","last_active":"2024-03-25T02:35:00.425759+0000","last_peered":"2024-03-25T02:35:00.425759+0000","last_clean":"2024-03-25T02:35:00.425759+0000","last_became_active":"2024-03-25T02:01:38.008862+0000","last_became_peered":"2024-03-25T02:01:38.008862+0000","last_unstale":"2024-03-25T02:35:00.425759+0000","last_undegraded":"2024-03-25T02:35:00.425759+0000","last_fullsized":"2024-03-25T02:35:00.425759+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:48:26.588270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.624056+0000","last_change":"2024-03-25T02:01:55.459478+0000","last_active":"2024-03-25T02:35:01.624056+0000","last_peered":"2024-03-25T02:35:01.624056+0000","last_clean":"2024-03-25T02:35:01.624056+0000","last_became_active":"2024-03-25T02:01:55.452605+0000","last_became_peered":"2024-03-25T02:01:55.452605+0000","last_unstale":"2024-03-25T02:35:01.624056+0000","last_undegraded":"2024-03-25T02:35:01.624056+0000","last_fullsized":"2024-03-25T02:35:01.624056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:52:09.879541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"23'77","reported_seq":2503,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.347895+0000","last_change":"2024-03-25T02:00:26.482050+0000","last_active":"2024-03-25T02:34:58.347895+0000","last_peered":"2024-03-25T02:34:58.347895+0000","last_clean":"2024-03-25T02:34:58.347895+0000","last_became_active":"2024-03-25T02:00:25.840109+0000","last_became_peered":"2024-03-25T02:00:25.840109+0000","last_unstale":"2024-03-25T02:34:58.347895+0000","last_undegraded":"2024-03-25T02:34:58.347895+0000","last_fullsized":"2024-03-25T02:34:58.347895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_clean_scrub_stamp":"2024-03-25T02:00:15.236797+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:57:04.634251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.551661+0000","last_change":"2024-03-25T02:00:33.873073+0000","last_active":"2024-03-25T02:35:01.551661+0000","last_peered":"2024-03-25T02:35:01.551661+0000","last_clean":"2024-03-25T02:35:01.551661+0000","last_became_active":"2024-03-25T02:00:33.865870+0000","last_became_peered":"2024-03-25T02:00:33.865870+0000","last_unstale":"2024-03-25T02:35:01.551661+0000","last_undegraded":"2024-03-25T02:35:01.551661+0000","last_fullsized":"2024-03-25T02:35:01.551661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:52:25.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.625707+0000","last_change":"2024-03-25T02:01:42.911218+0000","last_active":"2024-03-25T02:35:01.625707+0000","last_peered":"2024-03-25T02:35:01.625707+0000","last_clean":"2024-03-25T02:35:01.625707+0000","last_became_active":"2024-03-25T02:01:42.904032+0000","last_became_peered":"2024-03-25T02:01:42.904032+0000","last_unstale":"2024-03-25T02:35:01.625707+0000","last_undegraded":"2024-03-25T02:35:01.625707+0000","last_fullsized":"2024-03-25T02:35:01.625707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:28:21.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.511421+0000","last_change":"2024-03-25T02:00:33.383977+0000","last_active":"2024-03-25T02:35:01.511421+0000","last_peered":"2024-03-25T02:35:01.511421+0000","last_clean":"2024-03-25T02:35:01.511421+0000","last_became_active":"2024-03-25T02:00:33.356691+0000","last_became_peered":"2024-03-25T02:00:33.356691+0000","last_unstale":"2024-03-25T02:35:01.511421+0000","last_undegraded":"2024-03-25T02:35:01.511421+0000","last_fullsized":"2024-03-25T02:35:01.511421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:57.213604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2406,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.700884+0000","last_change":"2024-03-25T02:01:39.128793+0000","last_active":"2024-03-25T02:35:01.700884+0000","last_peered":"2024-03-25T02:35:01.700884+0000","last_clean":"2024-03-25T02:35:01.700884+0000","last_became_active":"2024-03-25T02:01:39.122271+0000","last_became_peered":"2024-03-25T02:01:39.122271+0000","last_unstale":"2024-03-25T02:35:01.700884+0000","last_undegraded":"2024-03-25T02:35:01.700884+0000","last_fullsized":"2024-03-25T02:35:01.700884+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:57:32.356420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2419,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.344989+0000","last_change":"2024-03-25T02:00:33.459985+0000","last_active":"2024-03-25T02:35:03.344989+0000","last_peered":"2024-03-25T02:35:03.344989+0000","last_clean":"2024-03-25T02:35:03.344989+0000","last_became_active":"2024-03-25T02:00:33.429925+0000","last_became_peered":"2024-03-25T02:00:33.429925+0000","last_unstale":"2024-03-25T02:35:03.344989+0000","last_undegraded":"2024-03-25T02:35:03.344989+0000","last_fullsized":"2024-03-25T02:35:03.344989+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:09:49.004802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2398,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.356194+0000","last_change":"2024-03-25T02:01:53.314119+0000","last_active":"2024-03-25T02:34:58.356194+0000","last_peered":"2024-03-25T02:34:58.356194+0000","last_clean":"2024-03-25T02:34:58.356194+0000","last_became_active":"2024-03-25T02:01:53.306467+0000","last_became_peered":"2024-03-25T02:01:53.306467+0000","last_unstale":"2024-03-25T02:34:58.356194+0000","last_undegraded":"2024-03-25T02:34:58.356194+0000","last_fullsized":"2024-03-25T02:34:58.356194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:56:31.011637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.557938+0000","last_change":"2024-03-25T02:00:33.707296+0000","last_active":"2024-03-25T02:35:01.557938+0000","last_peered":"2024-03-25T02:35:01.557938+0000","last_clean":"2024-03-25T02:35:01.557938+0000","last_became_active":"2024-03-25T02:00:33.691900+0000","last_became_peered":"2024-03-25T02:00:33.691900+0000","last_unstale":"2024-03-25T02:35:01.557938+0000","last_undegraded":"2024-03-25T02:35:01.557938+0000","last_fullsized":"2024-03-25T02:35:01.557938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:26:57.139959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2406,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.676770+0000","last_change":"2024-03-25T02:01:39.897197+0000","last_active":"2024-03-25T02:35:01.676770+0000","last_peered":"2024-03-25T02:35:01.676770+0000","last_clean":"2024-03-25T02:35:01.676770+0000","last_became_active":"2024-03-25T02:01:39.890974+0000","last_became_peered":"2024-03-25T02:01:39.890974+0000","last_unstale":"2024-03-25T02:35:01.676770+0000","last_undegraded":"2024-03-25T02:35:01.676770+0000","last_fullsized":"2024-03-25T02:35:01.676770+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:25:57.375716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.706846+0000","last_change":"2024-03-25T02:00:33.702006+0000","last_active":"2024-03-25T02:35:01.706846+0000","last_peered":"2024-03-25T02:35:01.706846+0000","last_clean":"2024-03-25T02:35:01.706846+0000","last_became_active":"2024-03-25T02:00:33.693877+0000","last_became_peered":"2024-03-25T02:00:33.693877+0000","last_unstale":"2024-03-25T02:35:01.706846+0000","last_undegraded":"2024-03-25T02:35:01.706846+0000","last_fullsized":"2024-03-25T02:35:01.706846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:52:31.586659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.296971+0000","last_change":"2024-03-25T02:01:50.169634+0000","last_active":"2024-03-25T02:34:59.296971+0000","last_peered":"2024-03-25T02:34:59.296971+0000","last_clean":"2024-03-25T02:34:59.296971+0000","last_became_active":"2024-03-25T02:01:50.160922+0000","last_became_peered":"2024-03-25T02:01:50.160922+0000","last_unstale":"2024-03-25T02:34:59.296971+0000","last_undegraded":"2024-03-25T02:34:59.296971+0000","last_fullsized":"2024-03-25T02:34:59.296971+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:28:52.038897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.571339+0000","last_change":"2024-03-25T02:00:33.514196+0000","last_active":"2024-03-25T02:35:01.571339+0000","last_peered":"2024-03-25T02:35:01.571339+0000","last_clean":"2024-03-25T02:35:01.571339+0000","last_became_active":"2024-03-25T02:00:33.508976+0000","last_became_peered":"2024-03-25T02:00:33.508976+0000","last_unstale":"2024-03-25T02:35:01.571339+0000","last_undegraded":"2024-03-25T02:35:01.571339+0000","last_fullsized":"2024-03-25T02:35:01.571339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:14:11.807682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.300682+0000","last_change":"2024-03-25T02:01:59.814614+0000","last_active":"2024-03-25T02:34:59.300682+0000","last_peered":"2024-03-25T02:34:59.300682+0000","last_clean":"2024-03-25T02:34:59.300682+0000","last_became_active":"2024-03-25T02:01:59.807793+0000","last_became_peered":"2024-03-25T02:01:59.807793+0000","last_unstale":"2024-03-25T02:34:59.300682+0000","last_undegraded":"2024-03-25T02:34:59.300682+0000","last_fullsized":"2024-03-25T02:34:59.300682+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:42:56.823793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2421,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:02.589473+0000","last_change":"2024-03-25T02:00:33.937499+0000","last_active":"2024-03-25T02:35:02.589473+0000","last_peered":"2024-03-25T02:35:02.589473+0000","last_clean":"2024-03-25T02:35:02.589473+0000","last_became_active":"2024-03-25T02:00:33.909846+0000","last_became_peered":"2024-03-25T02:00:33.909846+0000","last_unstale":"2024-03-25T02:35:02.589473+0000","last_undegraded":"2024-03-25T02:35:02.589473+0000","last_fullsized":"2024-03-25T02:35:02.589473+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:26:26.798933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.569217+0000","last_change":"2024-03-25T02:01:37.864558+0000","last_active":"2024-03-25T02:35:01.569217+0000","last_peered":"2024-03-25T02:35:01.569217+0000","last_clean":"2024-03-25T02:35:01.569217+0000","last_became_active":"2024-03-25T02:01:37.856709+0000","last_became_peered":"2024-03-25T02:01:37.856709+0000","last_unstale":"2024-03-25T02:35:01.569217+0000","last_undegraded":"2024-03-25T02:35:01.569217+0000","last_fullsized":"2024-03-25T02:35:01.569217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:25:19.300671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2410,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.581481+0000","last_change":"2024-03-25T02:00:33.801095+0000","last_active":"2024-03-25T02:35:01.581481+0000","last_peered":"2024-03-25T02:35:01.581481+0000","last_clean":"2024-03-25T02:35:01.581481+0000","last_became_active":"2024-03-25T02:00:33.792626+0000","last_became_peered":"2024-03-25T02:00:33.792626+0000","last_unstale":"2024-03-25T02:35:01.581481+0000","last_undegraded":"2024-03-25T02:35:01.581481+0000","last_fullsized":"2024-03-25T02:35:01.581481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:02:09.659440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2408,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.363375+0000","last_change":"2024-03-25T02:01:54.523614+0000","last_active":"2024-03-25T02:35:03.363375+0000","last_peered":"2024-03-25T02:35:03.363375+0000","last_clean":"2024-03-25T02:35:03.363375+0000","last_became_active":"2024-03-25T02:01:54.518327+0000","last_became_peered":"2024-03-25T02:01:54.518327+0000","last_unstale":"2024-03-25T02:35:03.363375+0000","last_undegraded":"2024-03-25T02:35:03.363375+0000","last_fullsized":"2024-03-25T02:35:03.363375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:28:31.375645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.643728+0000","last_change":"2024-03-25T02:01:53.995287+0000","last_active":"2024-03-25T02:35:01.643728+0000","last_peered":"2024-03-25T02:35:01.643728+0000","last_clean":"2024-03-25T02:35:01.643728+0000","last_became_active":"2024-03-25T02:01:53.989515+0000","last_became_peered":"2024-03-25T02:01:53.989515+0000","last_unstale":"2024-03-25T02:35:01.643728+0000","last_undegraded":"2024-03-25T02:35:01.643728+0000","last_fullsized":"2024-03-25T02:35:01.643728+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:43:06.616302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.647118+0000","last_change":"2024-03-25T02:01:55.429116+0000","last_active":"2024-03-25T02:35:01.647118+0000","last_peered":"2024-03-25T02:35:01.647118+0000","last_clean":"2024-03-25T02:35:01.647118+0000","last_became_active":"2024-03-25T02:01:55.419583+0000","last_became_peered":"2024-03-25T02:01:55.419583+0000","last_unstale":"2024-03-25T02:35:01.647118+0000","last_undegraded":"2024-03-25T02:35:01.647118+0000","last_fullsized":"2024-03-25T02:35:01.647118+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:54:42.345480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.704922+0000","last_change":"2024-03-25T02:01:56.827003+0000","last_active":"2024-03-25T02:35:01.704922+0000","last_peered":"2024-03-25T02:35:01.704922+0000","last_clean":"2024-03-25T02:35:01.704922+0000","last_became_active":"2024-03-25T02:01:56.799197+0000","last_became_peered":"2024-03-25T02:01:56.799197+0000","last_unstale":"2024-03-25T02:35:01.704922+0000","last_undegraded":"2024-03-25T02:35:01.704922+0000","last_fullsized":"2024-03-25T02:35:01.704922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:49:03.924799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2401,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.632313+0000","last_change":"2024-03-25T02:01:50.245600+0000","last_active":"2024-03-25T02:35:01.632313+0000","last_peered":"2024-03-25T02:35:01.632313+0000","last_clean":"2024-03-25T02:35:01.632313+0000","last_became_active":"2024-03-25T02:01:50.226409+0000","last_became_peered":"2024-03-25T02:01:50.226409+0000","last_unstale":"2024-03-25T02:35:01.632313+0000","last_undegraded":"2024-03-25T02:35:01.632313+0000","last_fullsized":"2024-03-25T02:35:01.632313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:33:00.764629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2397,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.354373+0000","last_change":"2024-03-25T02:01:39.447936+0000","last_active":"2024-03-25T02:34:58.354373+0000","last_peered":"2024-03-25T02:34:58.354373+0000","last_clean":"2024-03-25T02:34:58.354373+0000","last_became_active":"2024-03-25T02:01:39.437348+0000","last_became_peered":"2024-03-25T02:01:39.437348+0000","last_unstale":"2024-03-25T02:34:58.354373+0000","last_undegraded":"2024-03-25T02:34:58.354373+0000","last_fullsized":"2024-03-25T02:34:58.354373+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:31:57.241440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.285476+0000","last_change":"2024-03-25T02:01:43.296920+0000","last_active":"2024-03-25T02:34:59.285476+0000","last_peered":"2024-03-25T02:34:59.285476+0000","last_clean":"2024-03-25T02:34:59.285476+0000","last_became_active":"2024-03-25T02:01:43.285633+0000","last_became_peered":"2024-03-25T02:01:43.285633+0000","last_unstale":"2024-03-25T02:34:59.285476+0000","last_undegraded":"2024-03-25T02:34:59.285476+0000","last_fullsized":"2024-03-25T02:34:59.285476+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:17:06.380180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2397,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.345296+0000","last_change":"2024-03-25T02:01:41.663556+0000","last_active":"2024-03-25T02:34:58.345296+0000","last_peered":"2024-03-25T02:34:58.345296+0000","last_clean":"2024-03-25T02:34:58.345296+0000","last_became_active":"2024-03-25T02:01:41.631996+0000","last_became_peered":"2024-03-25T02:01:41.631996+0000","last_unstale":"2024-03-25T02:34:58.345296+0000","last_undegraded":"2024-03-25T02:34:58.345296+0000","last_fullsized":"2024-03-25T02:34:58.345296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:56:07.666478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.545302+0000","last_change":"2024-03-25T02:01:44.130872+0000","last_active":"2024-03-25T02:35:01.545302+0000","last_peered":"2024-03-25T02:35:01.545302+0000","last_clean":"2024-03-25T02:35:01.545302+0000","last_became_active":"2024-03-25T02:01:44.123357+0000","last_became_peered":"2024-03-25T02:01:44.123357+0000","last_unstale":"2024-03-25T02:35:01.545302+0000","last_undegraded":"2024-03-25T02:35:01.545302+0000","last_fullsized":"2024-03-25T02:35:01.545302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:47:38.979247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2434,"reported_epoch":1246,"state":"active+clean","last_fresh":"2024-03-25T02:35:00.433598+0000","last_change":"2024-03-25T02:01:40.283761+0000","last_active":"2024-03-25T02:35:00.433598+0000","last_peered":"2024-03-25T02:35:00.433598+0000","last_clean":"2024-03-25T02:35:00.433598+0000","last_became_active":"2024-03-25T02:01:40.275307+0000","last_became_peered":"2024-03-25T02:01:40.275307+0000","last_unstale":"2024-03-25T02:35:00.433598+0000","last_undegraded":"2024-03-25T02:35:00.433598+0000","last_fullsized":"2024-03-25T02:35:00.433598+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:28.413233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.295133+0000","last_change":"2024-03-25T02:01:50.267025+0000","last_active":"2024-03-25T02:34:59.295133+0000","last_peered":"2024-03-25T02:34:59.295133+0000","last_clean":"2024-03-25T02:34:59.295133+0000","last_became_active":"2024-03-25T02:01:50.240201+0000","last_became_peered":"2024-03-25T02:01:50.240201+0000","last_unstale":"2024-03-25T02:34:59.295133+0000","last_undegraded":"2024-03-25T02:34:59.295133+0000","last_fullsized":"2024-03-25T02:34:59.295133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:56.028592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.703082+0000","last_change":"2024-03-25T02:01:56.819977+0000","last_active":"2024-03-25T02:35:01.703082+0000","last_peered":"2024-03-25T02:35:01.703082+0000","last_clean":"2024-03-25T02:35:01.703082+0000","last_became_active":"2024-03-25T02:01:56.787332+0000","last_became_peered":"2024-03-25T02:01:56.787332+0000","last_unstale":"2024-03-25T02:35:01.703082+0000","last_undegraded":"2024-03-25T02:35:01.703082+0000","last_fullsized":"2024-03-25T02:35:01.703082+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:10:10.439975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2402,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.565652+0000","last_change":"2024-03-25T02:01:40.340111+0000","last_active":"2024-03-25T02:35:01.565652+0000","last_peered":"2024-03-25T02:35:01.565652+0000","last_clean":"2024-03-25T02:35:01.565652+0000","last_became_active":"2024-03-25T02:01:40.334290+0000","last_became_peered":"2024-03-25T02:01:40.334290+0000","last_unstale":"2024-03-25T02:35:01.565652+0000","last_undegraded":"2024-03-25T02:35:01.565652+0000","last_fullsized":"2024-03-25T02:35:01.565652+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:51.615630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":20,"seq":85899346334,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70872,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300968,"statfs":{"total":96636764160,"available":96564191232,"internally_reserved":0,"allocated":3674112,"data_stored":1502020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346340,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71248,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300592,"statfs":{"total":96636764160,"available":96563806208,"internally_reserved":0,"allocated":4063232,"data_stored":1891806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379042,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70900,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300940,"statfs":{"total":96636764160,"available":96564162560,"internally_reserved":0,"allocated":3702784,"data_stored":1517254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379044,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66736,"kb_used_data":3612,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305104,"statfs":{"total":96636764160,"available":96568426496,"internally_reserved":0,"allocated":3698688,"data_stored":1517249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71276,"kb_used_data":4004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300564,"statfs":{"total":96636764160,"available":96563777536,"internally_reserved":0,"allocated":4100096,"data_stored":1915089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66736,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305104,"statfs":{"total":96636764160,"available":96568426496,"internally_reserved":0,"allocated":3702784,"data_stored":1521345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70872,"kb_used_data":3600,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300968,"statfs":{"total":96636764160,"available":96564191232,"internally_reserved":0,"allocated":3686400,"data_stored":1509825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542564,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70908,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300932,"statfs":{"total":96636764160,"available":96564154368,"internally_reserved":0,"allocated":3719168,"data_stored":1524914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-25T02:35:05.804 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-25T02:35:05.804 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-25T02:35:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.915+0000 7fe4812fb640 10 monclient: get_monmap_and_config 2024-03-25T02:35:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.915+0000 7fe4812fb640 10 monclient: build_initial_monmap 2024-03-25T02:35:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.915+0000 7fe4812fb640 10 monclient: monmap: 2024-03-25T02:35:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:05.904 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:05.916851+0000 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:05.916851+0000 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:05.905 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.915+0000 7fe4812fb640 1 Processor -- start 2024-03-25T02:35:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 1 -- start start 2024-03-25T02:35:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient: init 2024-03-25T02:35:05.907 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe47c0a9b70 0x7fe47c1437c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe47c0a9b70 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe47c0b33f0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c14da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe47c0a9f50 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe47c0abae0 con 0x7fe47c0b33f0 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe47c0abdc0 con 0x7fe47c0a9b70 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fe47c0ac0a0 con 0x7fe47c0a9f50 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.916+0000 7fe4812fb640 10 monclient(hunting): authenticate will time out at 3425.106591s 2024-03-25T02:35:05.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47affd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe47c0a9b70 0x7fe47c1437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47b7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47b7fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c14da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50552/0 (socket says 172.21.15.83:50552) 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47b7fe640 1 -- 172.21.15.83:0/4259619410 learned_addr learned my addr 172.21.15.83:0/4259619410 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47b7fe640 10 monclient(hunting): get_auth_request con 0x7fe47c0a9f50 auth_method 0 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47affd640 10 monclient(hunting): get_auth_request con 0x7fe47c0a9b70 auth_method 0 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.917+0000 7fe47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.918+0000 7fe47b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.918+0000 7fe47b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.918+0000 7fe47b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.918+0000 7fe47affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.918+0000 7fe47affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.918+0000 7fe47affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 10 monclient(hunting): handle_auth_done global_id 17216 payload 274 2024-03-25T02:35:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 1 -- 172.21.15.83:0/4259619410 >> v2:172.21.15.83:3300/0 conn(0x7fe47c0a9b70 msgr2=0x7fe47c1437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:05.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 1 --2- 172.21.15.83:0/4259619410 >> v2:172.21.15.83:3300/0 conn(0x7fe47c0a9b70 0x7fe47c1437c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 1 -- 172.21.15.83:0/4259619410 >> v1:172.21.15.83:6789/0 conn(0x7fe47c0b33f0 legacy=0x7fe47c140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 10 monclient: _finish_hunting 0 2024-03-25T02:35:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 1 monclient: found mon.noname-b 2024-03-25T02:35:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.920+0000 7fe47b7fe640 1 -- 172.21.15.83:0/4259619410 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47c1432a0 con 0x7fe47c0a9f50 2024-03-25T02:35:05.911 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.922+0000 7fe47b7fe640 1 --2- 172.21.15.83:0/4259619410 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c14da10 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7fe474009e50 tx=0x7fe47402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71465dbbc05008c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe479ffb640 1 -- 172.21.15.83:0/4259619410 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47402cb00 con 0x7fe47c0a9f50 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe479ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe479ffb640 10 monclient: dump: 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.913 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.914 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.914 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe479ffb640 10 monclient: _finish_auth 0 2024-03-25T02:35:05.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe479ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:35:05.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe479ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:05.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe4812fb640 5 monclient: authenticate success, global_id 17216 2024-03-25T02:35:05.914 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.924+0000 7fe4812fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe479ffb640 1 -- 172.21.15.83:0/4259619410 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe47402cca0 con 0x7fe47c0a9f50 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe479ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe479ffb640 1 -- 172.21.15.83:0/4259619410 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe474034610 con 0x7fe47c0a9f50 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe4812fb640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe4812fb640 4 set_mon_vals no callback set 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe479ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe479ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.925+0000 7fe479ffb640 10 monclient: dump: 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.915 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.916 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.916 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 10 monclient: shutdown 2024-03-25T02:35:05.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:05.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 -- 172.21.15.83:0/4259619410 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 msgr2=0x7fe47c14da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 --2- 172.21.15.83:0/4259619410 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c14da10 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7fe474009e50 tx=0x7fe47402bf60 comp rx=0 tx=0).stop 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 -- 172.21.15.83:0/4259619410 shutdown_connections 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 --2- 172.21.15.83:0/4259619410 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c14da10 unknown :-1 s=CLOSED pgs=3195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 --2- 172.21.15.83:0/4259619410 >> v2:172.21.15.83:3300/0 conn(0x7fe47c0a9b70 0x7fe47c1437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 -- 172.21.15.83:0/4259619410 >> 172.21.15.83:0/4259619410 conn(0x7fe47c067e30 msgr2=0x7fe47c0b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 -- 172.21.15.83:0/4259619410 shutdown_connections 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.926+0000 7fe4812fb640 1 -- 172.21.15.83:0/4259619410 wait complete. 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.927+0000 7fe4812fb640 10 monclient: build_initial_monmap 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.927+0000 7fe4812fb640 10 monclient: monmap: 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:05.920 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.927+0000 7fe4812fb640 1 librados: starting msgr at 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.927+0000 7fe4812fb640 1 librados: starting objecter 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.927+0000 7fe4812fb640 1 Processor -- start 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 -- start start 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 librados: setting wanted keys 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 librados: calling monclient init 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient: init 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:35:05.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0a9b70 0x7fe47c0b87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe47c0a9b70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c1464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe47c0a9f50 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe47c0b33f0 0x7fe47c0b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe47c0b33f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c143d00 con 0x7fe47c0b33f0 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe47c140cc0 con 0x7fe47c0a9b70 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fe47c143e80 con 0x7fe47c0a9f50 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.928+0000 7fe4812fb640 10 monclient(hunting): authenticate will time out at 3425.118663s 2024-03-25T02:35:05.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe47c0b33f0 0x7fe47c0b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47a7fc640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c1464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe47c0b33f0 0x7fe47c0b7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49462/0 (socket says 172.21.15.83:49462) 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47a7fc640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c1464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50568/0 (socket says 172.21.15.83:50568) 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47b7fe640 1 -- 172.21.15.83:0/831478845 learned_addr learned my addr 172.21.15.83:0/831478845 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47b7fe640 10 monclient(hunting): get_auth_request con 0x7fe47c0b33f0 auth_method 0 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47affd640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0a9b70 0x7fe47c0b87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47a7fc640 10 monclient(hunting): get_auth_request con 0x7fe47c0a9f50 auth_method 0 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.923 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47affd640 10 monclient(hunting): get_auth_request con 0x7fe47c0a9b70 auth_method 0 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.929+0000 7fe47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:05.924 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.930+0000 7fe47affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:05.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 10 monclient(hunting): handle_auth_done global_id 17219 payload 824 2024-03-25T02:35:05.925 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 1 -- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0a9b70 msgr2=0x7fe47c0b87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0a9b70 0x7fe47c0b87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 1 -- 172.21.15.83:0/831478845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe47c0b33f0 msgr2=0x7fe47c0b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe47c0b33f0 0x7fe47c0b7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 10 monclient: _finish_hunting 0 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 1 monclient: found mon.noname-c 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.933+0000 7fe47a7fc640 1 -- 172.21.15.83:0/831478845 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47c0b92e0 con 0x7fe47c0a9f50 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.936+0000 7fe47a7fc640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c1464c0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00c340 tx=0x7fe46c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.936+0000 7fe4637fe640 10 client.?.objecter ms_handle_connect 0x7fe47c0a9f50 2024-03-25T02:35:05.926 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.936+0000 7fe4637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 1 -- 172.21.15.83:0/831478845 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c0052d0 con 0x7fe47c0a9f50 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: dump: 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.930 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: _finish_auth 0 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: _check_auth_tickets 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 1 -- 172.21.15.83:0/831478845 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe46c005490 con 0x7fe47c0a9f50 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 5 monclient: authenticate success, global_id 17219 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 10 monclient: _renew_subs 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47c1d8a10 con 0x7fe47c0a9f50 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:05.931 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 1 -- 172.21.15.83:0/831478845 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c007260 con 0x7fe47c0a9f50 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4637fe640 10 monclient: dump: 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4797fa640 4 set_mon_vals no callback set 2024-03-25T02:35:05.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 10 client.17219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 10 monclient: _renew_subs 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe47c1d8f10 con 0x7fe47c0a9f50 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.940+0000 7fe4812fb640 1 librados: init done 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.941+0000 7fe4812fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.941+0000 7fe4812fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.941+0000 7fe4812fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:05.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:05.941+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe47c144200 con 0x7fe47c0a9f50 2024-03-25T02:35:06.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.196+0000 7fe4637fe640 1 -- 172.21.15.83:0/831478845 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fe46c007400 con 0x7fe47c0a9f50 2024-03-25T02:35:06.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.197+0000 7fe4637fe640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe46806da70 0x7fe46806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:06.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.197+0000 7fe47affd640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe46806da70 0x7fe46806ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:06.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.197+0000 7fe47affd640 10 monclient: get_auth_request con 0x7fe46806da70 auth_method 0 2024-03-25T02:35:06.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.198+0000 7fe47affd640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe46806da70 0x7fe46806ff30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe470009aa0 tx=0x7fe470009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:06.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.198+0000 7fe4637fe640 10 client.17219.objecter ms_handle_connect 0x7fe46806da70 2024-03-25T02:35:06.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 1 -- 172.21.15.83:0/831478845 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7fe46c056c00 con 0x7fe47c0a9f50 2024-03-25T02:35:06.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 10 client.17219.objecter ms_dispatch 0x7fe47c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:06.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 3 client.17219.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:06.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 3 client.17219.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:06.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 20 client.17219.objecter dump_active .. 0 homeless 2024-03-25T02:35:06.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 1 -- 172.21.15.83:0/831478845 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe47c144200 con 0x7fe47c0a9f50 2024-03-25T02:35:06.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:06.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.212+0000 7fe4637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:06.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.422+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe47c0ab240 con 0x7fe46806da70 2024-03-25T02:35:06.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.426+0000 7fe4637fe640 1 -- 172.21.15.83:0/831478845 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116240 (secure 0 0 0) 0x7fe47c0ab240 con 0x7fe46806da70 2024-03-25T02:35:06.419 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:35:06.419 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-25T02:35:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 10 librados: watch_flush enter 2024-03-25T02:35:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 10 librados: watch_flush exit 2024-03-25T02:35:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 20 client.17219.objecter shutdown clearing up homeless session... 2024-03-25T02:35:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 10 client.17219.objecter successfully canceled tick 2024-03-25T02:35:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe46806da70 msgr2=0x7fe46806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe46806da70 0x7fe46806ff30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe470009aa0 tx=0x7fe470009190 comp rx=0 tx=0).stop 2024-03-25T02:35:06.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 10 monclient: shutdown 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 msgr2=0x7fe47c1464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c1464c0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00c340 tx=0x7fe46c00c800 comp rx=0 tx=0).stop 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 shutdown_connections 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe46806da70 0x7fe46806ff30 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe47c0b33f0 0x7fe47c0b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe47c0a9f50 0x7fe47c1464c0 unknown :-1 s=CLOSED pgs=3196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 --2- 172.21.15.83:0/831478845 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe47c0a9b70 0x7fe47c0b87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 >> 172.21.15.83:0/831478845 conn(0x7fe47c067e30 msgr2=0x7fe47c13eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 shutdown_connections 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 -- 172.21.15.83:0/831478845 wait complete. 2024-03-25T02:35:06.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.428+0000 7fe4812fb640 1 librados: shutdown 2024-03-25T02:35:06.429 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":2299,"stamp":"2024-03-25T02:35:04.931421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":89,"num_write":121,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":559548,"kb_used_data":29636,"kb_used_omap":12,"kb_used_meta":529779,"kb_avail":754415172,"statfs":{"total":773094113280,"available":772521136128,"internally_reserved":0,"allocated":30347264,"data_stored":12899502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":542494256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":118,"apply_latency_ms":118,"commit_latency_ns":118000000,"apply_latency_ns":118000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000678"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2401,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.614411+0000","last_change":"2024-03-25T02:01:59.223551+0000","last_active":"2024-03-25T02:35:01.614411+0000","last_peered":"2024-03-25T02:35:01.614411+0000","last_clean":"2024-03-25T02:35:01.614411+0000","last_became_active":"2024-03-25T02:01:59.217800+0000","last_became_peered":"2024-03-25T02:01:59.217800+0000","last_unstale":"2024-03-25T02:35:01.614411+0000","last_undegraded":"2024-03-25T02:35:01.614411+0000","last_fullsized":"2024-03-25T02:35:01.614411+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:52:12.956708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.291634+0000","last_change":"2024-03-25T02:01:45.802865+0000","last_active":"2024-03-25T02:34:59.291634+0000","last_peered":"2024-03-25T02:34:59.291634+0000","last_clean":"2024-03-25T02:34:59.291634+0000","last_became_active":"2024-03-25T02:01:45.737540+0000","last_became_peered":"2024-03-25T02:01:45.737540+0000","last_unstale":"2024-03-25T02:34:59.291634+0000","last_undegraded":"2024-03-25T02:34:59.291634+0000","last_fullsized":"2024-03-25T02:34:59.291634+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:53:36.729760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2410,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:02.601950+0000","last_change":"2024-03-25T02:01:57.033633+0000","last_active":"2024-03-25T02:35:02.601950+0000","last_peered":"2024-03-25T02:35:02.601950+0000","last_clean":"2024-03-25T02:35:02.601950+0000","last_became_active":"2024-03-25T02:01:57.014842+0000","last_became_peered":"2024-03-25T02:01:57.014842+0000","last_unstale":"2024-03-25T02:35:02.601950+0000","last_undegraded":"2024-03-25T02:35:02.601950+0000","last_fullsized":"2024-03-25T02:35:02.601950+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:30:09.539808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.697841+0000","last_change":"2024-03-25T02:01:39.501653+0000","last_active":"2024-03-25T02:35:01.697841+0000","last_peered":"2024-03-25T02:35:01.697841+0000","last_clean":"2024-03-25T02:35:01.697841+0000","last_became_active":"2024-03-25T02:01:39.486234+0000","last_became_peered":"2024-03-25T02:01:39.486234+0000","last_unstale":"2024-03-25T02:35:01.697841+0000","last_undegraded":"2024-03-25T02:35:01.697841+0000","last_fullsized":"2024-03-25T02:35:01.697841+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:45:00.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.709178+0000","last_change":"2024-03-25T02:01:39.797384+0000","last_active":"2024-03-25T02:35:01.709178+0000","last_peered":"2024-03-25T02:35:01.709178+0000","last_clean":"2024-03-25T02:35:01.709178+0000","last_became_active":"2024-03-25T02:01:39.792053+0000","last_became_peered":"2024-03-25T02:01:39.792053+0000","last_unstale":"2024-03-25T02:35:01.709178+0000","last_undegraded":"2024-03-25T02:35:01.709178+0000","last_fullsized":"2024-03-25T02:35:01.709178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:38.937942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2409,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.365260+0000","last_change":"2024-03-25T02:01:42.883607+0000","last_active":"2024-03-25T02:35:03.365260+0000","last_peered":"2024-03-25T02:35:03.365260+0000","last_clean":"2024-03-25T02:35:03.365260+0000","last_became_active":"2024-03-25T02:01:42.877188+0000","last_became_peered":"2024-03-25T02:01:42.877188+0000","last_unstale":"2024-03-25T02:35:03.365260+0000","last_undegraded":"2024-03-25T02:35:03.365260+0000","last_fullsized":"2024-03-25T02:35:03.365260+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:55:25.547629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2409,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.298817+0000","last_change":"2024-03-25T02:01:45.794313+0000","last_active":"2024-03-25T02:34:59.298817+0000","last_peered":"2024-03-25T02:34:59.298817+0000","last_clean":"2024-03-25T02:34:59.298817+0000","last_became_active":"2024-03-25T02:01:45.785913+0000","last_became_peered":"2024-03-25T02:01:45.785913+0000","last_unstale":"2024-03-25T02:34:59.298817+0000","last_undegraded":"2024-03-25T02:34:59.298817+0000","last_fullsized":"2024-03-25T02:34:59.298817+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:46:04.871320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.302500+0000","last_change":"2024-03-25T02:01:47.960650+0000","last_active":"2024-03-25T02:34:59.302500+0000","last_peered":"2024-03-25T02:34:59.302500+0000","last_clean":"2024-03-25T02:34:59.302500+0000","last_became_active":"2024-03-25T02:01:47.939684+0000","last_became_peered":"2024-03-25T02:01:47.939684+0000","last_unstale":"2024-03-25T02:34:59.302500+0000","last_undegraded":"2024-03-25T02:34:59.302500+0000","last_fullsized":"2024-03-25T02:34:59.302500+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:51:58.276672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2424,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.342976+0000","last_change":"2024-03-25T02:01:58.313196+0000","last_active":"2024-03-25T02:35:03.342976+0000","last_peered":"2024-03-25T02:35:03.342976+0000","last_clean":"2024-03-25T02:35:03.342976+0000","last_became_active":"2024-03-25T02:01:58.306972+0000","last_became_peered":"2024-03-25T02:01:58.306972+0000","last_unstale":"2024-03-25T02:35:03.342976+0000","last_undegraded":"2024-03-25T02:35:03.342976+0000","last_fullsized":"2024-03-25T02:35:03.342976+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:05:07.200253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2408,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.661493+0000","last_change":"2024-03-25T02:01:44.749489+0000","last_active":"2024-03-25T02:35:01.661493+0000","last_peered":"2024-03-25T02:35:01.661493+0000","last_clean":"2024-03-25T02:35:01.661493+0000","last_became_active":"2024-03-25T02:01:44.731330+0000","last_became_peered":"2024-03-25T02:01:44.731330+0000","last_unstale":"2024-03-25T02:35:01.661493+0000","last_undegraded":"2024-03-25T02:35:01.661493+0000","last_fullsized":"2024-03-25T02:35:01.661493+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:40:48.117174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2437,"reported_epoch":1246,"state":"active+clean","last_fresh":"2024-03-25T02:35:00.425759+0000","last_change":"2024-03-25T02:01:38.014396+0000","last_active":"2024-03-25T02:35:00.425759+0000","last_peered":"2024-03-25T02:35:00.425759+0000","last_clean":"2024-03-25T02:35:00.425759+0000","last_became_active":"2024-03-25T02:01:38.008862+0000","last_became_peered":"2024-03-25T02:01:38.008862+0000","last_unstale":"2024-03-25T02:35:00.425759+0000","last_undegraded":"2024-03-25T02:35:00.425759+0000","last_fullsized":"2024-03-25T02:35:00.425759+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:48:26.588270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.624056+0000","last_change":"2024-03-25T02:01:55.459478+0000","last_active":"2024-03-25T02:35:01.624056+0000","last_peered":"2024-03-25T02:35:01.624056+0000","last_clean":"2024-03-25T02:35:01.624056+0000","last_became_active":"2024-03-25T02:01:55.452605+0000","last_became_peered":"2024-03-25T02:01:55.452605+0000","last_unstale":"2024-03-25T02:35:01.624056+0000","last_undegraded":"2024-03-25T02:35:01.624056+0000","last_fullsized":"2024-03-25T02:35:01.624056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:52:09.879541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"23'77","reported_seq":2503,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.347895+0000","last_change":"2024-03-25T02:00:26.482050+0000","last_active":"2024-03-25T02:34:58.347895+0000","last_peered":"2024-03-25T02:34:58.347895+0000","last_clean":"2024-03-25T02:34:58.347895+0000","last_became_active":"2024-03-25T02:00:25.840109+0000","last_became_peered":"2024-03-25T02:00:25.840109+0000","last_unstale":"2024-03-25T02:34:58.347895+0000","last_undegraded":"2024-03-25T02:34:58.347895+0000","last_fullsized":"2024-03-25T02:34:58.347895+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_clean_scrub_stamp":"2024-03-25T02:00:15.236797+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:57:04.634251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.551661+0000","last_change":"2024-03-25T02:00:33.873073+0000","last_active":"2024-03-25T02:35:01.551661+0000","last_peered":"2024-03-25T02:35:01.551661+0000","last_clean":"2024-03-25T02:35:01.551661+0000","last_became_active":"2024-03-25T02:00:33.865870+0000","last_became_peered":"2024-03-25T02:00:33.865870+0000","last_unstale":"2024-03-25T02:35:01.551661+0000","last_undegraded":"2024-03-25T02:35:01.551661+0000","last_fullsized":"2024-03-25T02:35:01.551661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:52:25.672032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2407,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.625707+0000","last_change":"2024-03-25T02:01:42.911218+0000","last_active":"2024-03-25T02:35:01.625707+0000","last_peered":"2024-03-25T02:35:01.625707+0000","last_clean":"2024-03-25T02:35:01.625707+0000","last_became_active":"2024-03-25T02:01:42.904032+0000","last_became_peered":"2024-03-25T02:01:42.904032+0000","last_unstale":"2024-03-25T02:35:01.625707+0000","last_undegraded":"2024-03-25T02:35:01.625707+0000","last_fullsized":"2024-03-25T02:35:01.625707+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:28:21.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2414,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.511421+0000","last_change":"2024-03-25T02:00:33.383977+0000","last_active":"2024-03-25T02:35:01.511421+0000","last_peered":"2024-03-25T02:35:01.511421+0000","last_clean":"2024-03-25T02:35:01.511421+0000","last_became_active":"2024-03-25T02:00:33.356691+0000","last_became_peered":"2024-03-25T02:00:33.356691+0000","last_unstale":"2024-03-25T02:35:01.511421+0000","last_undegraded":"2024-03-25T02:35:01.511421+0000","last_fullsized":"2024-03-25T02:35:01.511421+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:26:57.213604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2406,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.700884+0000","last_change":"2024-03-25T02:01:39.128793+0000","last_active":"2024-03-25T02:35:01.700884+0000","last_peered":"2024-03-25T02:35:01.700884+0000","last_clean":"2024-03-25T02:35:01.700884+0000","last_became_active":"2024-03-25T02:01:39.122271+0000","last_became_peered":"2024-03-25T02:01:39.122271+0000","last_unstale":"2024-03-25T02:35:01.700884+0000","last_undegraded":"2024-03-25T02:35:01.700884+0000","last_fullsized":"2024-03-25T02:35:01.700884+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:57:32.356420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2419,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.344989+0000","last_change":"2024-03-25T02:00:33.459985+0000","last_active":"2024-03-25T02:35:03.344989+0000","last_peered":"2024-03-25T02:35:03.344989+0000","last_clean":"2024-03-25T02:35:03.344989+0000","last_became_active":"2024-03-25T02:00:33.429925+0000","last_became_peered":"2024-03-25T02:00:33.429925+0000","last_unstale":"2024-03-25T02:35:03.344989+0000","last_undegraded":"2024-03-25T02:35:03.344989+0000","last_fullsized":"2024-03-25T02:35:03.344989+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:09:49.004802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2398,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.356194+0000","last_change":"2024-03-25T02:01:53.314119+0000","last_active":"2024-03-25T02:34:58.356194+0000","last_peered":"2024-03-25T02:34:58.356194+0000","last_clean":"2024-03-25T02:34:58.356194+0000","last_became_active":"2024-03-25T02:01:53.306467+0000","last_became_peered":"2024-03-25T02:01:53.306467+0000","last_unstale":"2024-03-25T02:34:58.356194+0000","last_undegraded":"2024-03-25T02:34:58.356194+0000","last_fullsized":"2024-03-25T02:34:58.356194+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:56:31.011637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.557938+0000","last_change":"2024-03-25T02:00:33.707296+0000","last_active":"2024-03-25T02:35:01.557938+0000","last_peered":"2024-03-25T02:35:01.557938+0000","last_clean":"2024-03-25T02:35:01.557938+0000","last_became_active":"2024-03-25T02:00:33.691900+0000","last_became_peered":"2024-03-25T02:00:33.691900+0000","last_unstale":"2024-03-25T02:35:01.557938+0000","last_undegraded":"2024-03-25T02:35:01.557938+0000","last_fullsized":"2024-03-25T02:35:01.557938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:26:57.139959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2406,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.676770+0000","last_change":"2024-03-25T02:01:39.897197+0000","last_active":"2024-03-25T02:35:01.676770+0000","last_peered":"2024-03-25T02:35:01.676770+0000","last_clean":"2024-03-25T02:35:01.676770+0000","last_became_active":"2024-03-25T02:01:39.890974+0000","last_became_peered":"2024-03-25T02:01:39.890974+0000","last_unstale":"2024-03-25T02:35:01.676770+0000","last_undegraded":"2024-03-25T02:35:01.676770+0000","last_fullsized":"2024-03-25T02:35:01.676770+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:25:57.375716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.706846+0000","last_change":"2024-03-25T02:00:33.702006+0000","last_active":"2024-03-25T02:35:01.706846+0000","last_peered":"2024-03-25T02:35:01.706846+0000","last_clean":"2024-03-25T02:35:01.706846+0000","last_became_active":"2024-03-25T02:00:33.693877+0000","last_became_peered":"2024-03-25T02:00:33.693877+0000","last_unstale":"2024-03-25T02:35:01.706846+0000","last_undegraded":"2024-03-25T02:35:01.706846+0000","last_fullsized":"2024-03-25T02:35:01.706846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:52:31.586659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.296971+0000","last_change":"2024-03-25T02:01:50.169634+0000","last_active":"2024-03-25T02:34:59.296971+0000","last_peered":"2024-03-25T02:34:59.296971+0000","last_clean":"2024-03-25T02:34:59.296971+0000","last_became_active":"2024-03-25T02:01:50.160922+0000","last_became_peered":"2024-03-25T02:01:50.160922+0000","last_unstale":"2024-03-25T02:34:59.296971+0000","last_undegraded":"2024-03-25T02:34:59.296971+0000","last_fullsized":"2024-03-25T02:34:59.296971+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:28:52.038897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2413,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.571339+0000","last_change":"2024-03-25T02:00:33.514196+0000","last_active":"2024-03-25T02:35:01.571339+0000","last_peered":"2024-03-25T02:35:01.571339+0000","last_clean":"2024-03-25T02:35:01.571339+0000","last_became_active":"2024-03-25T02:00:33.508976+0000","last_became_peered":"2024-03-25T02:00:33.508976+0000","last_unstale":"2024-03-25T02:35:01.571339+0000","last_undegraded":"2024-03-25T02:35:01.571339+0000","last_fullsized":"2024-03-25T02:35:01.571339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:14:11.807682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.300682+0000","last_change":"2024-03-25T02:01:59.814614+0000","last_active":"2024-03-25T02:34:59.300682+0000","last_peered":"2024-03-25T02:34:59.300682+0000","last_clean":"2024-03-25T02:34:59.300682+0000","last_became_active":"2024-03-25T02:01:59.807793+0000","last_became_peered":"2024-03-25T02:01:59.807793+0000","last_unstale":"2024-03-25T02:34:59.300682+0000","last_undegraded":"2024-03-25T02:34:59.300682+0000","last_fullsized":"2024-03-25T02:34:59.300682+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:42:56.823793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2421,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:02.589473+0000","last_change":"2024-03-25T02:00:33.937499+0000","last_active":"2024-03-25T02:35:02.589473+0000","last_peered":"2024-03-25T02:35:02.589473+0000","last_clean":"2024-03-25T02:35:02.589473+0000","last_became_active":"2024-03-25T02:00:33.909846+0000","last_became_peered":"2024-03-25T02:00:33.909846+0000","last_unstale":"2024-03-25T02:35:02.589473+0000","last_undegraded":"2024-03-25T02:35:02.589473+0000","last_fullsized":"2024-03-25T02:35:02.589473+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:26:26.798933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.569217+0000","last_change":"2024-03-25T02:01:37.864558+0000","last_active":"2024-03-25T02:35:01.569217+0000","last_peered":"2024-03-25T02:35:01.569217+0000","last_clean":"2024-03-25T02:35:01.569217+0000","last_became_active":"2024-03-25T02:01:37.856709+0000","last_became_peered":"2024-03-25T02:01:37.856709+0000","last_unstale":"2024-03-25T02:35:01.569217+0000","last_undegraded":"2024-03-25T02:35:01.569217+0000","last_fullsized":"2024-03-25T02:35:01.569217+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:25:19.300671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2410,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.581481+0000","last_change":"2024-03-25T02:00:33.801095+0000","last_active":"2024-03-25T02:35:01.581481+0000","last_peered":"2024-03-25T02:35:01.581481+0000","last_clean":"2024-03-25T02:35:01.581481+0000","last_became_active":"2024-03-25T02:00:33.792626+0000","last_became_peered":"2024-03-25T02:00:33.792626+0000","last_unstale":"2024-03-25T02:35:01.581481+0000","last_undegraded":"2024-03-25T02:35:01.581481+0000","last_fullsized":"2024-03-25T02:35:01.581481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:02:09.659440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2408,"reported_epoch":1248,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.363375+0000","last_change":"2024-03-25T02:01:54.523614+0000","last_active":"2024-03-25T02:35:03.363375+0000","last_peered":"2024-03-25T02:35:03.363375+0000","last_clean":"2024-03-25T02:35:03.363375+0000","last_became_active":"2024-03-25T02:01:54.518327+0000","last_became_peered":"2024-03-25T02:01:54.518327+0000","last_unstale":"2024-03-25T02:35:03.363375+0000","last_undegraded":"2024-03-25T02:35:03.363375+0000","last_fullsized":"2024-03-25T02:35:03.363375+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:28:31.375645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.643728+0000","last_change":"2024-03-25T02:01:53.995287+0000","last_active":"2024-03-25T02:35:01.643728+0000","last_peered":"2024-03-25T02:35:01.643728+0000","last_clean":"2024-03-25T02:35:01.643728+0000","last_became_active":"2024-03-25T02:01:53.989515+0000","last_became_peered":"2024-03-25T02:01:53.989515+0000","last_unstale":"2024-03-25T02:35:01.643728+0000","last_undegraded":"2024-03-25T02:35:01.643728+0000","last_fullsized":"2024-03-25T02:35:01.643728+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:43:06.616302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.647118+0000","last_change":"2024-03-25T02:01:55.429116+0000","last_active":"2024-03-25T02:35:01.647118+0000","last_peered":"2024-03-25T02:35:01.647118+0000","last_clean":"2024-03-25T02:35:01.647118+0000","last_became_active":"2024-03-25T02:01:55.419583+0000","last_became_peered":"2024-03-25T02:01:55.419583+0000","last_unstale":"2024-03-25T02:35:01.647118+0000","last_undegraded":"2024-03-25T02:35:01.647118+0000","last_fullsized":"2024-03-25T02:35:01.647118+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:54:42.345480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.704922+0000","last_change":"2024-03-25T02:01:56.827003+0000","last_active":"2024-03-25T02:35:01.704922+0000","last_peered":"2024-03-25T02:35:01.704922+0000","last_clean":"2024-03-25T02:35:01.704922+0000","last_became_active":"2024-03-25T02:01:56.799197+0000","last_became_peered":"2024-03-25T02:01:56.799197+0000","last_unstale":"2024-03-25T02:35:01.704922+0000","last_undegraded":"2024-03-25T02:35:01.704922+0000","last_fullsized":"2024-03-25T02:35:01.704922+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:49:03.924799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2401,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.632313+0000","last_change":"2024-03-25T02:01:50.245600+0000","last_active":"2024-03-25T02:35:01.632313+0000","last_peered":"2024-03-25T02:35:01.632313+0000","last_clean":"2024-03-25T02:35:01.632313+0000","last_became_active":"2024-03-25T02:01:50.226409+0000","last_became_peered":"2024-03-25T02:01:50.226409+0000","last_unstale":"2024-03-25T02:35:01.632313+0000","last_undegraded":"2024-03-25T02:35:01.632313+0000","last_fullsized":"2024-03-25T02:35:01.632313+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:33:00.764629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2397,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.354373+0000","last_change":"2024-03-25T02:01:39.447936+0000","last_active":"2024-03-25T02:34:58.354373+0000","last_peered":"2024-03-25T02:34:58.354373+0000","last_clean":"2024-03-25T02:34:58.354373+0000","last_became_active":"2024-03-25T02:01:39.437348+0000","last_became_peered":"2024-03-25T02:01:39.437348+0000","last_unstale":"2024-03-25T02:34:58.354373+0000","last_undegraded":"2024-03-25T02:34:58.354373+0000","last_fullsized":"2024-03-25T02:34:58.354373+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:31:57.241440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2408,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.285476+0000","last_change":"2024-03-25T02:01:43.296920+0000","last_active":"2024-03-25T02:34:59.285476+0000","last_peered":"2024-03-25T02:34:59.285476+0000","last_clean":"2024-03-25T02:34:59.285476+0000","last_became_active":"2024-03-25T02:01:43.285633+0000","last_became_peered":"2024-03-25T02:01:43.285633+0000","last_unstale":"2024-03-25T02:34:59.285476+0000","last_undegraded":"2024-03-25T02:34:59.285476+0000","last_fullsized":"2024-03-25T02:34:59.285476+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:17:06.380180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2397,"reported_epoch":1244,"state":"active+clean","last_fresh":"2024-03-25T02:34:58.345296+0000","last_change":"2024-03-25T02:01:41.663556+0000","last_active":"2024-03-25T02:34:58.345296+0000","last_peered":"2024-03-25T02:34:58.345296+0000","last_clean":"2024-03-25T02:34:58.345296+0000","last_became_active":"2024-03-25T02:01:41.631996+0000","last_became_peered":"2024-03-25T02:01:41.631996+0000","last_unstale":"2024-03-25T02:34:58.345296+0000","last_undegraded":"2024-03-25T02:34:58.345296+0000","last_fullsized":"2024-03-25T02:34:58.345296+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:56:07.666478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2403,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.545302+0000","last_change":"2024-03-25T02:01:44.130872+0000","last_active":"2024-03-25T02:35:01.545302+0000","last_peered":"2024-03-25T02:35:01.545302+0000","last_clean":"2024-03-25T02:35:01.545302+0000","last_became_active":"2024-03-25T02:01:44.123357+0000","last_became_peered":"2024-03-25T02:01:44.123357+0000","last_unstale":"2024-03-25T02:35:01.545302+0000","last_undegraded":"2024-03-25T02:35:01.545302+0000","last_fullsized":"2024-03-25T02:35:01.545302+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:47:38.979247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2434,"reported_epoch":1246,"state":"active+clean","last_fresh":"2024-03-25T02:35:00.433598+0000","last_change":"2024-03-25T02:01:40.283761+0000","last_active":"2024-03-25T02:35:00.433598+0000","last_peered":"2024-03-25T02:35:00.433598+0000","last_clean":"2024-03-25T02:35:00.433598+0000","last_became_active":"2024-03-25T02:01:40.275307+0000","last_became_peered":"2024-03-25T02:01:40.275307+0000","last_unstale":"2024-03-25T02:35:00.433598+0000","last_undegraded":"2024-03-25T02:35:00.433598+0000","last_fullsized":"2024-03-25T02:35:00.433598+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:02:28.413233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2407,"reported_epoch":1245,"state":"active+clean","last_fresh":"2024-03-25T02:34:59.295133+0000","last_change":"2024-03-25T02:01:50.267025+0000","last_active":"2024-03-25T02:34:59.295133+0000","last_peered":"2024-03-25T02:34:59.295133+0000","last_clean":"2024-03-25T02:34:59.295133+0000","last_became_active":"2024-03-25T02:01:50.240201+0000","last_became_peered":"2024-03-25T02:01:50.240201+0000","last_unstale":"2024-03-25T02:34:59.295133+0000","last_undegraded":"2024-03-25T02:34:59.295133+0000","last_fullsized":"2024-03-25T02:34:59.295133+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:56.028592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2404,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.703082+0000","last_change":"2024-03-25T02:01:56.819977+0000","last_active":"2024-03-25T02:35:01.703082+0000","last_peered":"2024-03-25T02:35:01.703082+0000","last_clean":"2024-03-25T02:35:01.703082+0000","last_became_active":"2024-03-25T02:01:56.787332+0000","last_became_peered":"2024-03-25T02:01:56.787332+0000","last_unstale":"2024-03-25T02:35:01.703082+0000","last_undegraded":"2024-03-25T02:35:01.703082+0000","last_fullsized":"2024-03-25T02:35:01.703082+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:10:10.439975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2402,"reported_epoch":1247,"state":"active+clean","last_fresh":"2024-03-25T02:35:01.565652+0000","last_change":"2024-03-25T02:01:40.340111+0000","last_active":"2024-03-25T02:35:01.565652+0000","last_peered":"2024-03-25T02:35:01.565652+0000","last_clean":"2024-03-25T02:35:01.565652+0000","last_became_active":"2024-03-25T02:01:40.334290+0000","last_became_peered":"2024-03-25T02:01:40.334290+0000","last_unstale":"2024-03-25T02:35:01.565652+0000","last_undegraded":"2024-03-25T02:35:01.565652+0000","last_fullsized":"2024-03-25T02:35:01.565652+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:11:51.615630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":20,"seq":85899346334,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70872,"kb_used_data":3588,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300968,"statfs":{"total":96636764160,"available":96564191232,"internally_reserved":0,"allocated":3674112,"data_stored":1502020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346340,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71248,"kb_used_data":3968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300592,"statfs":{"total":96636764160,"available":96563806208,"internally_reserved":0,"allocated":4063232,"data_stored":1891806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379042,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70900,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300940,"statfs":{"total":96636764160,"available":96564162560,"internally_reserved":0,"allocated":3702784,"data_stored":1517254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379044,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66736,"kb_used_data":3612,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305104,"statfs":{"total":96636764160,"available":96568426496,"internally_reserved":0,"allocated":3698688,"data_stored":1517249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71276,"kb_used_data":4004,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300564,"statfs":{"total":96636764160,"available":96563777536,"internally_reserved":0,"allocated":4100096,"data_stored":1915089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509857,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66736,"kb_used_data":3616,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305104,"statfs":{"total":96636764160,"available":96568426496,"internally_reserved":0,"allocated":3702784,"data_stored":1521345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70872,"kb_used_data":3600,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300968,"statfs":{"total":96636764160,"available":96564191232,"internally_reserved":0,"allocated":3686400,"data_stored":1509825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542564,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70908,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300932,"statfs":{"total":96636764160,"available":96564154368,"internally_reserved":0,"allocated":3719168,"data_stored":1524914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-25T02:35:06.431 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T02:35:06.534 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.545+0000 7f3e9be2c640 10 monclient: get_monmap_and_config 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.545+0000 7f3e9be2c640 10 monclient: build_initial_monmap 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.545+0000 7f3e9be2c640 10 monclient: monmap: 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:06.546598+0000 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:06.546598+0000 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:06.535 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:06.536 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:06.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.545+0000 7f3e9be2c640 1 Processor -- start 2024-03-25T02:35:06.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.545+0000 7f3e9be2c640 1 -- start start 2024-03-25T02:35:06.536 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.545+0000 7f3e9be2c640 10 monclient: init 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e940b04a0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3e940b2df0 0x7f3e94140400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient(hunting): picked mon.noname-c con 0x7f3e940b2df0 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e94147350 0x7f3e94140940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient(hunting): picked mon.noname-a con 0x7f3e94147350 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e940b2410 con 0x7f3e940b04a0 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f3e94140e80 con 0x7f3e94147350 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:06.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f3e940b26f0 con 0x7f3e940b2df0 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.546+0000 7f3e9be2c640 10 monclient(hunting): authenticate will time out at 3425.736515s 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e9a3c9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e94147350 0x7f3e94140940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e993c7640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f3e940b2df0 0x7f3e94140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e9a3c9640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f3e94147350 0x7f3e94140940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49476/0 (socket says 172.21.15.83:49476) 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e9a3c9640 1 -- 172.21.15.83:0/3038605434 learned_addr learned my addr 172.21.15.83:0/3038605434 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e9a3c9640 10 monclient(hunting): get_auth_request con 0x7f3e94147350 auth_method 0 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e9a3c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e9a3c9640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e9a3c9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e993c7640 10 monclient(hunting): get_auth_request con 0x7f3e940b2df0 auth_method 0 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e993c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e993c7640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:06.539 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.547+0000 7f3e993c7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.548+0000 7f3e993c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.548+0000 7f3e993c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.548+0000 7f3e993c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.548+0000 7f3e9a3c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.548+0000 7f3e9a3c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:06.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.548+0000 7f3e9a3c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 10 monclient(hunting): handle_auth_done global_id 22024 payload 274 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 1 -- 172.21.15.83:0/3038605434 >> v2:172.21.15.83:3300/0 conn(0x7f3e94147350 msgr2=0x7f3e94140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 1 --2- 172.21.15.83:0/3038605434 >> v2:172.21.15.83:3300/0 conn(0x7f3e94147350 0x7f3e94140940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 1 -- 172.21.15.83:0/3038605434 >> v1:172.21.15.83:6789/0 conn(0x7f3e940b04a0 legacy=0x7f3e94146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 10 monclient: _finish_hunting 0 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 1 monclient: found mon.noname-c 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.550+0000 7f3e993c7640 1 -- 172.21.15.83:0/3038605434 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e940b29d0 con 0x7f3e940b2df0 2024-03-25T02:35:06.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.552+0000 7f3e993c7640 1 --2- 172.21.15.83:0/3038605434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b2df0 0x7f3e94140400 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 crypto rx=0x7f3e840097a0 tx=0x7f3e8402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b59ec7213fce7bac server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 1 -- 172.21.15.83:0/3038605434 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84035020 con 0x7f3e940b2df0 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: dump: 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:06.546 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: _finish_auth 0 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: _check_auth_tickets 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 1 -- 172.21.15.83:0/3038605434 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e84031830 con 0x7f3e940b2df0 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 5 monclient: authenticate success, global_id 22024 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 1 -- 172.21.15.83:0/3038605434 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84031c00 con 0x7f3e940b2df0 2024-03-25T02:35:06.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 4 set_mon_vals no callback set 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e98bc6640 10 monclient: dump: 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:06.548 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 10 monclient: shutdown 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3038605434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b2df0 msgr2=0x7f3e94140400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3038605434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b2df0 0x7f3e94140400 secure :-1 s=READY pgs=5528 cs=0 l=1 rev1=1 crypto rx=0x7f3e840097a0 tx=0x7f3e8402b8d0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3038605434 shutdown_connections 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3038605434 >> v2:172.21.15.83:3300/0 conn(0x7f3e94147350 0x7f3e94140940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3038605434 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b2df0 0x7f3e94140400 unknown :-1 s=CLOSED pgs=5528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.556+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3038605434 >> 172.21.15.83:0/3038605434 conn(0x7f3e940b8a00 msgr2=0x7f3e940b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.557+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3038605434 shutdown_connections 2024-03-25T02:35:06.549 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.557+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3038605434 wait complete. 2024-03-25T02:35:06.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.557+0000 7f3e9be2c640 10 monclient: build_initial_monmap 2024-03-25T02:35:06.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.557+0000 7f3e9be2c640 10 monclient: monmap: 2024-03-25T02:35:06.552 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:06.552 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:06.552 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:06.552 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 librados: starting msgr at 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 librados: starting objecter 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 Processor -- start 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 -- start start 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 librados: setting wanted keys 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 librados: calling monclient init 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient: init 2024-03-25T02:35:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b04a0 0x7f3e94141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): picked mon.noname-b con 0x7f3e940b04a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3e940b2df0 0x7f3e94134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): picked mon.noname-c con 0x7f3e940b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e94147350 0x7f3e941350a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): picked mon.noname-a con 0x7f3e94147350 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e940b0a00 con 0x7f3e94147350 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e940b0880 con 0x7f3e940b04a0 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9414daf0 con 0x7f3e940b2df0 2024-03-25T02:35:06.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.558+0000 7f3e9be2c640 10 monclient(hunting): authenticate will time out at 3425.748692s 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e9a3c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e94147350 0x7f3e941350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e993c7640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3e940b2df0 0x7f3e94134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e99bc8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b04a0 0x7f3e94141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e9a3c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e94147350 0x7f3e941350a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49482/0 (socket says 172.21.15.83:49482) 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e9a3c9640 1 -- 172.21.15.83:0/3681501676 learned_addr learned my addr 172.21.15.83:0/3681501676 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e9a3c9640 10 monclient(hunting): get_auth_request con 0x7f3e94147350 auth_method 0 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e9a3c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e9a3c9640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e9a3c9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e993c7640 10 monclient(hunting): get_auth_request con 0x7f3e940b2df0 auth_method 0 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e993c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e993c7640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:06.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e993c7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e99bc8640 10 monclient(hunting): get_auth_request con 0x7f3e940b04a0 auth_method 0 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e99bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e99bc8640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.559+0000 7f3e99bc8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e99bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e99bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e99bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e9a3c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e9a3c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e9a3c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e993c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e993c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.560+0000 7f3e993c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:06.556 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 10 monclient(hunting): handle_auth_done global_id 22027 payload 824 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 1 -- 172.21.15.83:0/3681501676 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3e940b2df0 msgr2=0x7f3e94134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3e940b2df0 0x7f3e94134b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 1 -- 172.21.15.83:0/3681501676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e94147350 msgr2=0x7f3e941350a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e94147350 0x7f3e941350a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 10 monclient: _finish_hunting 0 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 1 monclient: found mon.noname-b 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:06.557 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.562+0000 7f3e99bc8640 1 -- 172.21.15.83:0/3681501676 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9413a2f0 con 0x7f3e940b04a0 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.565+0000 7f3e99bc8640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b04a0 0x7f3e94141000 secure :-1 s=READY pgs=5529 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000c340 tx=0x7f3e9000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.565+0000 7f3e8a7fc640 10 client.?.objecter ms_handle_connect 0x7f3e940b04a0 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.565+0000 7f3e8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.568+0000 7f3e8a7fc640 1 -- 172.21.15.83:0/3681501676 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e900052d0 con 0x7f3e940b04a0 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.568+0000 7f3e8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: dump: 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.559 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: _finish_auth 0 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 1 -- 172.21.15.83:0/3681501676 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e90005490 con 0x7f3e940b04a0 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 5 monclient: authenticate success, global_id 22027 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 monclient: _renew_subs 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 1 -- 172.21.15.83:0/3681501676 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90007260 con 0x7f3e940b04a0 2024-03-25T02:35:06.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9413a5d0 con 0x7f3e940b04a0 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8bfff640 4 set_mon_vals no callback set 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e8a7fc640 10 monclient: dump: 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:06.561 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 client.22027.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 monclient: _renew_subs 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e941355e0 con 0x7f3e940b04a0 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 1 librados: init done 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:06.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.569+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e94139260 con 0x7f3e940b04a0 2024-03-25T02:35:06.807 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.818+0000 7f3e8a7fc640 1 -- 172.21.15.83:0/3681501676 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f3e90007400 con 0x7f3e940b04a0 2024-03-25T02:35:06.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.819+0000 7f3e8a7fc640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3e7806da70 0x7f3e7806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:06.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.819+0000 7f3e993c7640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3e7806da70 0x7f3e7806ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:06.808 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.819+0000 7f3e993c7640 10 monclient: get_auth_request con 0x7f3e7806da70 auth_method 0 2024-03-25T02:35:06.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.819+0000 7f3e993c7640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3e7806da70 0x7f3e7806ff30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3e84036000 tx=0x7f3e84008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:06.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.819+0000 7f3e8a7fc640 10 client.22027.objecter ms_handle_connect 0x7f3e7806da70 2024-03-25T02:35:06.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.822+0000 7f3e8a7fc640 1 -- 172.21.15.83:0/3681501676 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f3e90056fb0 con 0x7f3e940b04a0 2024-03-25T02:35:06.811 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.822+0000 7f3e8a7fc640 10 client.22027.objecter ms_dispatch 0x7f3e94000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:06.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.822+0000 7f3e8a7fc640 3 client.22027.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:06.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.822+0000 7f3e8a7fc640 3 client.22027.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:06.812 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.822+0000 7f3e8a7fc640 20 client.22027.objecter dump_active .. 0 homeless 2024-03-25T02:35:06.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.833+0000 7f3e8a7fc640 1 -- 172.21.15.83:0/3681501676 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3e90058f10 con 0x7f3e940b04a0 2024-03-25T02:35:06.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.833+0000 7f3e8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:06.823 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:06.833+0000 7f3e8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:06.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.008+0000 7f3e9be2c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:35:06.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.008+0000 7f3e9be2c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:35:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.008+0000 7f3e9be2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:06.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.008+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e94075810 con 0x7f3e940b04a0 2024-03-25T02:35:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.048+0000 7f3e8a7fc640 1 -- 172.21.15.83:0/3681501676 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1250) ==== 74+0+17721 (secure 0 0 0) 0x7f3e9005cfb0 con 0x7f3e940b04a0 2024-03-25T02:35:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.048+0000 7f3e8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T02:35:07.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.048+0000 7f3e8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:35:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:35:07.039 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":1250,"fsid":"86338f78-de3e-4243-afbe-49cfeed24a78","created":"2024-03-25T01:59:28.494253+0000","modified":"2024-03-25T02:35:03.768956+0000","last_up_change":"2024-03-25T02:00:23.723108+0000","last_in_change":"2024-03-25T01:59:34.053585+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":114,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-25T02:00:14.575846+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-25T02:00:31.420756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-03-25T02:01:34.188007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c3639213-7917-42d1-9609-bdc85e49f7bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6801","nonce":2678950219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6803","nonce":2678950219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6807","nonce":2678950219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":2678950219},{"type":"v1","addr":"172.21.15.83:6805","nonce":2678950219}]},"public_addr":"172.21.15.83:6801/2678950219","cluster_addr":"172.21.15.83:6803/2678950219","heartbeat_back_addr":"172.21.15.83:6807/2678950219","heartbeat_front_addr":"172.21.15.83:6805/2678950219","state":["exists","up"]},{"osd":1,"uuid":"8c5bf12d-d066-4fe7-a189-11c1d757e4b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1232,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6825","nonce":2597989034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6827","nonce":2597989034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6831","nonce":2597989034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":2597989034},{"type":"v1","addr":"172.21.15.83:6829","nonce":2597989034}]},"public_addr":"172.21.15.83:6825/2597989034","cluster_addr":"172.21.15.83:6827/2597989034","heartbeat_back_addr":"172.21.15.83:6831/2597989034","heartbeat_front_addr":"172.21.15.83:6829/2597989034","state":["exists","up"]},{"osd":2,"uuid":"19c0b51e-bebb-48b0-a930-bb1c181de85c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1239,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6817","nonce":602002618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6819","nonce":602002618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6823","nonce":602002618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":602002618},{"type":"v1","addr":"172.21.15.83:6821","nonce":602002618}]},"public_addr":"172.21.15.83:6817/602002618","cluster_addr":"172.21.15.83:6819/602002618","heartbeat_back_addr":"172.21.15.83:6823/602002618","heartbeat_front_addr":"172.21.15.83:6821/602002618","state":["exists","up"]},{"osd":3,"uuid":"415b88b9-7d5a-4162-a686-902197aca9fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6809","nonce":3432501411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6811","nonce":3432501411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6815","nonce":3432501411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3432501411},{"type":"v1","addr":"172.21.15.83:6813","nonce":3432501411}]},"public_addr":"172.21.15.83:6809/3432501411","cluster_addr":"172.21.15.83:6811/3432501411","heartbeat_back_addr":"172.21.15.83:6815/3432501411","heartbeat_front_addr":"172.21.15.83:6813/3432501411","state":["exists","up"]},{"osd":4,"uuid":"675b8bea-3596-4192-9556-b246aacefe85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6819","nonce":2579144563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6821","nonce":2579144563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6825","nonce":2579144563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2579144563},{"type":"v1","addr":"172.21.15.153:6823","nonce":2579144563}]},"public_addr":"172.21.15.153:6819/2579144563","cluster_addr":"172.21.15.153:6821/2579144563","heartbeat_back_addr":"172.21.15.153:6825/2579144563","heartbeat_front_addr":"172.21.15.153:6823/2579144563","state":["exists","up"]},{"osd":5,"uuid":"74d4302c-0ae2-4627-a3a9-f4e37ad1ae50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6811","nonce":2458137144}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6813","nonce":2458137144}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6817","nonce":2458137144}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2458137144},{"type":"v1","addr":"172.21.15.153:6815","nonce":2458137144}]},"public_addr":"172.21.15.153:6811/2458137144","cluster_addr":"172.21.15.153:6813/2458137144","heartbeat_back_addr":"172.21.15.153:6817/2458137144","heartbeat_front_addr":"172.21.15.153:6815/2458137144","state":["exists","up"]},{"osd":6,"uuid":"b754b718-408c-4a47-8d2c-a66d50ed18e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6803","nonce":4205760170}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6805","nonce":4205760170}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6809","nonce":4205760170}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":4205760170},{"type":"v1","addr":"172.21.15.153:6807","nonce":4205760170}]},"public_addr":"172.21.15.153:6803/4205760170","cluster_addr":"172.21.15.153:6805/4205760170","heartbeat_back_addr":"172.21.15.153:6809/4205760170","heartbeat_front_addr":"172.21.15.153:6807/4205760170","state":["exists","up"]},{"osd":7,"uuid":"23e89d51-cb05-4ca8-a5f0-1bf573aad7be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1246,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6827","nonce":2580901001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6829","nonce":2580901001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6833","nonce":2580901001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2580901001},{"type":"v1","addr":"172.21.15.153:6831","nonce":2580901001}]},"public_addr":"172.21.15.153:6827/2580901001","cluster_addr":"172.21.15.153:6829/2580901001","heartbeat_back_addr":"172.21.15.153:6833/2580901001","heartbeat_front_addr":"172.21.15.153:6831/2580901001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.217280+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.797510+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:12.322329+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:11.630439+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.336792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:09.768567+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.625955+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-25T02:00:10.445663+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":4,"length":3}]},{"pool":17,"snaps":[{"begin":4,"length":1}]},{"pool":25,"snaps":[{"begin":1,"length":1}]},{"pool":77,"snaps":[{"begin":1,"length":1}]},{"pool":92,"snaps":[{"begin":1,"length":1}]},{"pool":97,"snaps":[{"begin":7,"length":1}]},{"pool":103,"snaps":[{"begin":1,"length":2}]},{"pool":114,"snaps":[{"begin":3,"length":1}]},{"pool":122,"snaps":[{"begin":3,"length":2}]},{"pool":124,"snaps":[{"begin":2,"length":2}]},{"pool":131,"snaps":[{"begin":2,"length":3}]},{"pool":134,"snaps":[{"begin":2,"length":1}]},{"pool":137,"snaps":[{"begin":4,"length":2}]},{"pool":139,"snaps":[{"begin":1,"length":4}]},{"pool":140,"snaps":[{"begin":2,"length":2}]},{"pool":145,"snaps":[{"begin":4,"length":4}]},{"pool":146,"snaps":[{"begin":2,"length":1}]},{"pool":209,"snaps":[{"begin":7,"length":1}]},{"pool":219,"snaps":[{"begin":8,"length":1}]},{"pool":224,"snaps":[{"begin":10,"length":1}]},{"pool":227,"snaps":[{"begin":11,"length":2}]},{"pool":239,"snaps":[{"begin":14,"length":1}]},{"pool":282,"snaps":[{"begin":2,"length":1}]},{"pool":283,"snaps":[{"begin":3,"length":1}]},{"pool":286,"snaps":[{"begin":4,"length":1}]},{"pool":290,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 10 librados: watch_flush enter 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 10 librados: watch_flush exit 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 20 client.22027.objecter shutdown clearing up homeless session... 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 10 client.22027.objecter successfully canceled tick 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3e7806da70 msgr2=0x7f3e7806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3e7806da70 0x7f3e7806ff30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3e84036000 tx=0x7f3e84008030 comp rx=0 tx=0).stop 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 10 monclient: shutdown 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b04a0 msgr2=0x7f3e94141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b04a0 0x7f3e94141000 secure :-1 s=READY pgs=5529 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000c340 tx=0x7f3e9000c800 comp rx=0 tx=0).stop 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 shutdown_connections 2024-03-25T02:35:07.041 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f3e7806da70 0x7f3e7806ff30 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e94147350 0x7f3e941350a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f3e940b2df0 0x7f3e94134b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 --2- 172.21.15.83:0/3681501676 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e940b04a0 0x7f3e94141000 secure :-1 s=CLOSED pgs=5529 cs=0 l=1 rev1=1 crypto rx=0x7f3e9000c340 tx=0x7f3e9000c800 comp rx=0 tx=0).stop 2024-03-25T02:35:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 >> 172.21.15.83:0/3681501676 conn(0x7f3e940b8a00 msgr2=0x7f3e941401d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 shutdown_connections 2024-03-25T02:35:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 -- 172.21.15.83:0/3681501676 wait complete. 2024-03-25T02:35:07.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:07.051+0000 7f3e9be2c640 1 librados: shutdown 2024-03-25T02:35:08.052 INFO:tasks.ceph:Scrubbing osd.0 2024-03-25T02:35:08.053 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:08.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.169+0000 7f679998f640 10 monclient: get_monmap_and_config 2024-03-25T02:35:08.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.169+0000 7f679998f640 10 monclient: build_initial_monmap 2024-03-25T02:35:08.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.169+0000 7f679998f640 10 monclient: monmap: 2024-03-25T02:35:08.158 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:08.158 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:08.170829+0000 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:08.170829+0000 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:08.159 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 1 Processor -- start 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 1 -- start start 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient: init 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67940a9c10 0x7f67940a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient(hunting): picked mon.noname-a con 0x7f67940a9c10 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:08.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6794143760 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f67941409d0 0x7f6794143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient(hunting): picked mon.noname-c con 0x7f67941409d0 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67940abb80 con 0x7f6794143760 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f6794140de0 con 0x7f67940a9c10 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f67940abe60 con 0x7f67941409d0 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.170+0000 7f679998f640 10 monclient(hunting): authenticate will time out at 3427.360800s 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f6792ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67940a9c10 0x7f67940a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f67937fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f67941409d0 0x7f6794143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f6792ffd640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f67940a9c10 0x7f67940a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49492/0 (socket says 172.21.15.83:49492) 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f6792ffd640 1 -- 172.21.15.83:0/3288136563 learned_addr learned my addr 172.21.15.83:0/3288136563 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:08.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f6792ffd640 10 monclient(hunting): get_auth_request con 0x7f67940a9c10 auth_method 0 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f6792ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f6792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.171+0000 7f6792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f67937fe640 10 monclient(hunting): get_auth_request con 0x7f67941409d0 auth_method 0 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f67937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f67937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f67937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f6792ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f6792ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f6792ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.172+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 10 monclient(hunting): handle_auth_done global_id 15363 payload 274 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 1 -- 172.21.15.83:0/3288136563 >> v2:172.21.15.153:3300/0 conn(0x7f67941409d0 msgr2=0x7f6794143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 1 --2- 172.21.15.83:0/3288136563 >> v2:172.21.15.153:3300/0 conn(0x7f67941409d0 0x7f6794143bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 1 -- 172.21.15.83:0/3288136563 >> v1:172.21.15.153:6789/0 conn(0x7f6794143760 legacy=0x7f67941402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:08.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:35:08.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 1 monclient: found mon.noname-a 2024-03-25T02:35:08.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:08.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:08.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.174+0000 7f6792ffd640 1 -- 172.21.15.83:0/3288136563 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67940ac140 con 0x7f67940a9c10 2024-03-25T02:35:08.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.176+0000 7f6792ffd640 1 --2- 172.21.15.83:0/3288136563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67940a9c10 0x7f67940a9ff0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f6784009870 tx=0x7f678402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38b0767eafccf5c server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:08.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 1 -- 172.21.15.83:0/3288136563 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784035020 con 0x7f67940a9c10 2024-03-25T02:35:08.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:08.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: dump: 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: _finish_auth 0 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: _check_auth_tickets 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:08.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 1 -- 172.21.15.83:0/3288136563 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6784031870 con 0x7f67940a9c10 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f679998f640 5 monclient: authenticate success, global_id 15363 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f679998f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 1 -- 172.21.15.83:0/3288136563 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784031c40 con 0x7f67940a9c10 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f679998f640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f679998f640 4 set_mon_vals no callback set 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.179+0000 7f67927fc640 10 monclient: dump: 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.170 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.171 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.171 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.171 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.171 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.171 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 10 monclient: shutdown 2024-03-25T02:35:08.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:08.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 -- 172.21.15.83:0/3288136563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67940a9c10 msgr2=0x7f67940a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 --2- 172.21.15.83:0/3288136563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67940a9c10 0x7f67940a9ff0 secure :-1 s=READY pgs=3196 cs=0 l=1 rev1=1 crypto rx=0x7f6784009870 tx=0x7f678402b910 comp rx=0 tx=0).stop 2024-03-25T02:35:08.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 -- 172.21.15.83:0/3288136563 shutdown_connections 2024-03-25T02:35:08.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 --2- 172.21.15.83:0/3288136563 >> v2:172.21.15.153:3300/0 conn(0x7f67941409d0 0x7f6794143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 --2- 172.21.15.83:0/3288136563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67940a9c10 0x7f67940a9ff0 unknown :-1 s=CLOSED pgs=3196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 -- 172.21.15.83:0/3288136563 >> 172.21.15.83:0/3288136563 conn(0x7f67940b5560 msgr2=0x7f67940b5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 -- 172.21.15.83:0/3288136563 shutdown_connections 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.180+0000 7f679998f640 1 -- 172.21.15.83:0/3288136563 wait complete. 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 10 monclient: build_initial_monmap 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 10 monclient: monmap: 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:08.175 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 1 librados: starting msgr at 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 1 librados: starting objecter 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 1 Processor -- start 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 1 -- start start 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 1 librados: setting wanted keys 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 1 librados: calling monclient init 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.181+0000 7f679998f640 10 monclient: init 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f67940a9c10 0x7f67941360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): picked mon.noname-c con 0x7f67940a9c10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67941409d0 0x7f6794136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): picked mon.noname-b con 0x7f67941409d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6794143760 0x7f67941dc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): picked mon.noname-a con 0x7f6794143760 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67940b35e0 con 0x7f6794143760 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67940b3460 con 0x7f67941409d0 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f679414dab0 con 0x7f67940a9c10 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f679998f640 10 monclient(hunting): authenticate will time out at 3427.372293s 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f6792ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f67940a9c10 0x7f67941360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.182+0000 7f678bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67941409d0 0x7f6794136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f67937fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6794143760 0x7f67941dc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67941409d0 0x7f6794136fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:59126/0 (socket says 172.21.15.83:59126) 2024-03-25T02:35:08.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 1 -- 172.21.15.83:0/3951516390 learned_addr learned my addr 172.21.15.83:0/3951516390 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f67940a9c10 0x7f67941360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50592/0 (socket says 172.21.15.83:50592) 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 10 monclient(hunting): get_auth_request con 0x7f67941409d0 auth_method 0 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 10 monclient(hunting): get_auth_request con 0x7f67940a9c10 auth_method 0 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f67937fe640 10 monclient(hunting): get_auth_request con 0x7f6794143760 auth_method 0 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f67937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f67937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f67937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f678bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.183+0000 7f6792ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.184+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.184+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.184+0000 7f67937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 10 monclient(hunting): handle_auth_done global_id 17225 payload 824 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 1 -- 172.21.15.83:0/3951516390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67941409d0 msgr2=0x7f6794136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67941409d0 0x7f6794136fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 1 -- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6794143760 msgr2=0x7f67941dc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6794143760 0x7f67941dc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 10 monclient: _finish_hunting 0 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 1 monclient: found mon.noname-c 2024-03-25T02:35:08.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.186+0000 7f6792ffd640 1 -- 172.21.15.83:0/3951516390 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67941dbbe0 con 0x7f67940a9c10 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.188+0000 7f6792ffd640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f67940a9c10 0x7f67941360f0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f6784033000 tx=0x7f67840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.188+0000 7f6790ff9640 10 client.?.objecter ms_handle_connect 0x7f67940a9c10 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.188+0000 7f6790ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784040070 con 0x7f67940a9c10 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f6790ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f6790ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f6790ff9640 10 monclient: dump: 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.182 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f6790ff9640 10 monclient: _finish_auth 0 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f6790ff9640 10 monclient: _check_auth_tickets 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f6790ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 5 monclient: authenticate success, global_id 17225 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 10 monclient: _renew_subs 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6794136630 con 0x7f67940a9c10 2024-03-25T02:35:08.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 10 client.17225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 10 monclient: _renew_subs 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67941338f0 con 0x7f67940a9c10 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.190+0000 7f679998f640 1 librados: init done 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.191+0000 7f679998f640 10 client.17225.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.191+0000 7f679998f640 15 client.17225.objecter _session_command_op_assign -1 1 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.191+0000 7f679998f640 20 client.17225.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.191+0000 7f679998f640 10 client.17225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.191+0000 7f679998f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:08.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.191+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6794076d20 con 0x7f67940a9c10 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.193+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6784031870 con 0x7f67940a9c10 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.193+0000 7f6790ff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.193+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784035040 con 0x7f67940a9c10 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.193+0000 7f6790ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.193+0000 7f6790ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.193+0000 7f6790ff9640 10 monclient: dump: 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.185 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.186 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.193+0000 7f6791ffb640 4 set_mon_vals no callback set 2024-03-25T02:35:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.213+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f678404b060 con 0x7f67940a9c10 2024-03-25T02:35:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.213+0000 7f6790ff9640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677406da20 0x7f677406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.213+0000 7f678bfff640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677406da20 0x7f677406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.214+0000 7f678bfff640 10 monclient: get_auth_request con 0x7f677406da20 auth_method 0 2024-03-25T02:35:08.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.214+0000 7f678bfff640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677406da20 0x7f677406fee0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6794135bc0 tx=0x7f677c009110 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:08.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.214+0000 7f6790ff9640 10 client.17225.objecter ms_handle_connect 0x7f677406da20 2024-03-25T02:35:08.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f6784079520 con 0x7f67940a9c10 2024-03-25T02:35:08.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 10 client.17225.objecter ms_dispatch 0x7f6794000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:08.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 3 client.17225.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 3 client.17225.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 10 client.17225.objecter checking command 1 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7f6774075720 0x7f6774077bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 20 client.17225.objecter _get_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 20 client.17225.objecter put_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 15 client.17225.objecter _session_command_op_remove -1 1 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 20 client.17225.objecter _get_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 20 client.17225.objecter get_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 15 client.17225.objecter _session_command_op_assign 0 1 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 20 client.17225.objecter put_session s=0x7f6774075050 osd=0 4 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 10 client.17225.objecter _send_command 1 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 --> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f678403de50 con 0x7f6774075720 2024-03-25T02:35:08.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 20 client.17225.objecter dump_active .. 0 homeless 2024-03-25T02:35:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_get_version_reply(handle=1 version=1250) ==== 24+0+0 (secure 0 0 0) 0x7f67840777a0 con 0x7f67940a9c10 2024-03-25T02:35:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.217+0000 7f6790ff9640 10 monclient: handle_get_version_reply finishing 1 version 1250 2024-03-25T02:35:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.218+0000 7f67937fe640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7f6774075720 0x7f6774077bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.218+0000 7f67937fe640 10 monclient: get_auth_request con 0x7f6774075720 auth_method 0 2024-03-25T02:35:08.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.221+0000 7f67937fe640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7f6774075720 0x7f6774077bc0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:08.210 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.221+0000 7f6790ff9640 10 client.17225.objecter ms_handle_connect 0x7f6774075720 2024-03-25T02:35:08.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.244+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== osd.0 v2:172.21.15.83:6800/2678950219 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f678403de50 con 0x7f6774075720 2024-03-25T02:35:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.244+0000 7f6790ff9640 10 client.17225.objecter ms_dispatch 0x7f6794000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.244+0000 7f6790ff9640 10 client.17225.objecter _finish_command 1 = system:0 2024-03-25T02:35:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.244+0000 7f6790ff9640 20 client.17225.objecter put_session s=0x7f6774075050 osd=0 4 2024-03-25T02:35:08.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.244+0000 7f6790ff9640 15 client.17225.objecter _session_command_op_remove 0 1 2024-03-25T02:35:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 10 client.17225.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 15 client.17225.objecter _session_command_op_assign -1 2 2024-03-25T02:35:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 20 client.17225.objecter _get_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 20 client.17225.objecter put_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 20 client.17225.objecter _get_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 15 client.17225.objecter _session_command_op_remove -1 2 2024-03-25T02:35:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 20 client.17225.objecter get_session s=0x7f6774075050 osd=0 3 2024-03-25T02:35:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 15 client.17225.objecter _session_command_op_assign 0 2 2024-03-25T02:35:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 20 client.17225.objecter put_session s=0x7f6774075050 osd=0 4 2024-03-25T02:35:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 10 client.17225.objecter _send_command 2 2024-03-25T02:35:08.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.256+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 --> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6794075810 con 0x7f6774075720 2024-03-25T02:35:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.269+0000 7f6790ff9640 1 -- 172.21.15.83:0/3951516390 <== osd.0 v2:172.21.15.83:6800/2678950219 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6794075810 con 0x7f6774075720 2024-03-25T02:35:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.269+0000 7f6790ff9640 10 client.17225.objecter ms_dispatch 0x7f6794000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.269+0000 7f6790ff9640 10 client.17225.objecter _finish_command 2 = system:0 2024-03-25T02:35:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.269+0000 7f6790ff9640 20 client.17225.objecter put_session s=0x7f6774075050 osd=0 4 2024-03-25T02:35:08.258 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.269+0000 7f6790ff9640 15 client.17225.objecter _session_command_op_remove 0 2 2024-03-25T02:35:08.259 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:08.259 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:08.259 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:08.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.269+0000 7f679998f640 10 librados: watch_flush enter 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.269+0000 7f679998f640 10 librados: watch_flush exit 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 10 client.17225.objecter close_session for osd.0 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7f6774075720 msgr2=0x7f6774077bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:6800/2678950219,v1:172.21.15.83:6801/2678950219] conn(0x7f6774075720 0x7f6774077bc0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 20 client.17225.objecter put_session s=0x7f6774075050 osd=0 1 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 20 client.17225.objecter shutdown clearing up homeless session... 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 10 client.17225.objecter successfully canceled tick 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677406da20 msgr2=0x7f677406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f677406da20 0x7f677406fee0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6794135bc0 tx=0x7f677c009110 comp rx=0 tx=0).stop 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 10 monclient: shutdown 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f67940a9c10 msgr2=0x7f67941360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 --2- 172.21.15.83:0/3951516390 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f67940a9c10 0x7f67941360f0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f6784033000 tx=0x7f67840046c0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f67937fe640 1 -- 172.21.15.83:0/3951516390 reap_dead start 2024-03-25T02:35:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 shutdown_connections 2024-03-25T02:35:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 >> 172.21.15.83:0/3951516390 conn(0x7f67940b5560 msgr2=0x7f67940b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 shutdown_connections 2024-03-25T02:35:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 -- 172.21.15.83:0/3951516390 wait complete. 2024-03-25T02:35:08.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.270+0000 7f679998f640 1 librados: shutdown 2024-03-25T02:35:08.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.384+0000 7f303cd39640 10 monclient: get_monmap_and_config 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.384+0000 7f303cd39640 10 monclient: build_initial_monmap 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.384+0000 7f303cd39640 10 monclient: monmap: 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:08.385526+0000 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:08.385526+0000 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:08.373 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.374 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:08.374 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:08.374 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:08.374 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:08.374 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:08.374 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.384+0000 7f303cd39640 1 Processor -- start 2024-03-25T02:35:08.375 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.384+0000 7f303cd39640 1 -- start start 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.384+0000 7f303cd39640 10 monclient: init 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30380a9b70 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30380b33f0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f30380a9f50 0x7f303814ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient(hunting): picked mon.noname-a con 0x7f30380a9f50 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30380abae0 con 0x7f30380b33f0 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30380abdc0 con 0x7f30380a9b70 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f30380ac0a0 con 0x7f30380a9f50 2024-03-25T02:35:08.376 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:08.377 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.385+0000 7f303cd39640 10 monclient(hunting): authenticate will time out at 3427.575451s 2024-03-25T02:35:08.396 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.407+0000 7f3036d9d640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f30380a9f50 0x7f303814ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.408+0000 7f302ffff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f30380b33f0 0x7f3038140320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54180/0 (socket says 172.21.15.83:54180) 2024-03-25T02:35:08.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.408+0000 7f302ffff640 1 -- 172.21.15.83:0/3710921730 learned_addr learned my addr 172.21.15.83:0/3710921730 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:08.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.408+0000 7f3036d9d640 10 monclient(hunting): get_auth_request con 0x7f30380a9f50 auth_method 0 2024-03-25T02:35:08.397 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.408+0000 7f3036d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.408+0000 7f3036d9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.408+0000 7f3036d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.409+0000 7f3036d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.409+0000 7f3036d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.398 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.409+0000 7f3036d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 10 monclient(hunting): handle_auth_done global_id 15369 payload 274 2024-03-25T02:35:08.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 1 -- 172.21.15.83:0/3710921730 >> v1:172.21.15.153:6789/0 conn(0x7f30380a9b70 legacy=0x7f30381437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:08.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 1 -- 172.21.15.83:0/3710921730 >> v1:172.21.15.83:6789/0 conn(0x7f30380b33f0 legacy=0x7f3038140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 10 monclient: _finish_hunting 0 2024-03-25T02:35:08.400 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 1 monclient: found mon.noname-a 2024-03-25T02:35:08.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:08.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:08.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.411+0000 7f3036d9d640 1 -- 172.21.15.83:0/3710921730 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30381432a0 con 0x7f30380a9f50 2024-03-25T02:35:08.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.415+0000 7f3036d9d640 1 --2- 172.21.15.83:0/3710921730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f303814ea20 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f3030009d90 tx=0x7f303002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f6df6cdbb2c691 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f3035d9b640 1 -- 172.21.15.83:0/3710921730 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030039070 con 0x7f30380a9f50 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f3035d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f3035d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f3035d9b640 10 monclient: dump: 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.410 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f3035d9b640 10 monclient: _finish_auth 0 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f3035d9b640 10 monclient: _check_auth_tickets 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f3035d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f303cd39640 5 monclient: authenticate success, global_id 15369 2024-03-25T02:35:08.411 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.421+0000 7f303cd39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:08.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f3035d9b640 1 -- 172.21.15.83:0/3710921730 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303002cc60 con 0x7f30380a9f50 2024-03-25T02:35:08.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f3035d9b640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:08.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f3035d9b640 1 -- 172.21.15.83:0/3710921730 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030034660 con 0x7f30380a9f50 2024-03-25T02:35:08.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f3035d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.412 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f3035d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f3035d9b640 10 monclient: dump: 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f303cd39640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.422+0000 7f303cd39640 4 set_mon_vals no callback set 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 10 monclient: shutdown 2024-03-25T02:35:08.413 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 1 -- 172.21.15.83:0/3710921730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 msgr2=0x7f303814ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 1 --2- 172.21.15.83:0/3710921730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f303814ea20 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f3030009d90 tx=0x7f303002beb0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 1 -- 172.21.15.83:0/3710921730 shutdown_connections 2024-03-25T02:35:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 1 --2- 172.21.15.83:0/3710921730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f303814ea20 unknown :-1 s=CLOSED pgs=3198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 1 -- 172.21.15.83:0/3710921730 >> 172.21.15.83:0/3710921730 conn(0x7f3038067e30 msgr2=0x7f30380b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 1 -- 172.21.15.83:0/3710921730 shutdown_connections 2024-03-25T02:35:08.414 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.423+0000 7f303cd39640 1 -- 172.21.15.83:0/3710921730 wait complete. 2024-03-25T02:35:08.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 10 monclient: build_initial_monmap 2024-03-25T02:35:08.417 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 10 monclient: monmap: 2024-03-25T02:35:08.417 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:08.417 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:08.417 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:08.417 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:08.417 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 1 librados: starting msgr at 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 1 librados: starting objecter 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 1 Processor -- start 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 1 -- start start 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 1 librados: setting wanted keys 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 1 librados: calling monclient init 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.424+0000 7f303cd39640 10 monclient: init 2024-03-25T02:35:08.418 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f30380a9b70 0x7f3038142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): picked mon.noname-c con 0x7f30380a9b70 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f30381cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): picked mon.noname-a con 0x7f30380a9f50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30380b33f0 0x7f3038145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): picked mon.noname-b con 0x7f30380b33f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3038140bb0 con 0x7f30380a9f50 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3038140a30 con 0x7f30380b33f0 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f3038140d30 con 0x7f30380a9b70 2024-03-25T02:35:08.419 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303cd39640 10 monclient(hunting): authenticate will time out at 3427.615451s 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f302ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f30381cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.425+0000 7f303659c640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f30380a9b70 0x7f3038142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f3036d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30380b33f0 0x7f3038145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f30381cc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49516/0 (socket says 172.21.15.83:49516) 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 1 -- 172.21.15.83:0/365897153 learned_addr learned my addr 172.21.15.83:0/365897153 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 10 monclient(hunting): get_auth_request con 0x7f30380a9f50 auth_method 0 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f303659c640 10 monclient(hunting): get_auth_request con 0x7f30380a9b70 auth_method 0 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f303659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f303659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.420 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f303659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f3036d9d640 10 monclient(hunting): get_auth_request con 0x7f30380b33f0 auth_method 0 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f3036d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f3036d9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f3036d9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f302ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f303659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f303659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.426+0000 7f303659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.427+0000 7f3036d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.427+0000 7f3036d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:08.421 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.427+0000 7f3036d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:08.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 10 monclient(hunting): handle_auth_done global_id 15375 payload 824 2024-03-25T02:35:08.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 1 -- 172.21.15.83:0/365897153 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f30380a9b70 msgr2=0x7f3038142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.422 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f30380a9b70 0x7f3038142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 1 -- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30380b33f0 msgr2=0x7f3038145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30380b33f0 0x7f3038145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:08.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 1 monclient: found mon.noname-a 2024-03-25T02:35:08.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:08.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:08.423 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.431+0000 7f302ffff640 1 -- 172.21.15.83:0/365897153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3038142e90 con 0x7f30380a9f50 2024-03-25T02:35:08.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.434+0000 7f302ffff640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f30381cc4c0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f302000c400 tx=0x7f302000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:08.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.434+0000 7f302effd640 10 client.?.objecter ms_handle_connect 0x7f30380a9f50 2024-03-25T02:35:08.424 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.434+0000 7f302effd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:08.427 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 1 -- 172.21.15.83:0/365897153 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3020017070 con 0x7f30380a9f50 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: dump: 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.428 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: _finish_auth 0 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 1 -- 172.21.15.83:0/365897153 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3020005ce0 con 0x7f30380a9f50 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 1 -- 172.21.15.83:0/365897153 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30200072f0 con 0x7f30380a9f50 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f303cd39640 5 monclient: authenticate success, global_id 15375 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f303cd39640 10 monclient: _renew_subs 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f303cd39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30381cd380 con 0x7f30380a9f50 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f303559a640 4 set_mon_vals no callback set 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.437+0000 7f302effd640 10 monclient: dump: 2024-03-25T02:35:08.429 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 10 client.15375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 10 monclient: _renew_subs 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3038145d70 con 0x7f30380a9f50 2024-03-25T02:35:08.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 1 librados: init done 2024-03-25T02:35:08.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:08.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:08.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:08.431 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.438+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3038075810 con 0x7f30380a9f50 2024-03-25T02:35:08.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.458+0000 7f302effd640 1 -- 172.21.15.83:0/365897153 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f302001a070 con 0x7f30380a9f50 2024-03-25T02:35:08.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.459+0000 7f302effd640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f301806da70 0x7f301806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:08.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.459+0000 7f303659c640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f301806da70 0x7f301806ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:08.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.459+0000 7f303659c640 10 monclient: get_auth_request con 0x7f301806da70 auth_method 0 2024-03-25T02:35:08.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.459+0000 7f303659c640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f301806da70 0x7f301806ff30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3028006fd0 tx=0x7f3028009060 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:08.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.459+0000 7f302effd640 10 client.15375.objecter ms_handle_connect 0x7f301806da70 2024-03-25T02:35:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 1 -- 172.21.15.83:0/365897153 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f3020056ba0 con 0x7f30380a9f50 2024-03-25T02:35:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 10 client.15375.objecter ms_dispatch 0x7f3038000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 3 client.15375.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:08.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 3 client.15375.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 20 client.15375.objecter dump_active .. 0 homeless 2024-03-25T02:35:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 1 -- 172.21.15.83:0/365897153 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3020021020 con 0x7f30380a9f50 2024-03-25T02:35:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:08.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.704+0000 7f302effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:08.858 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:51.706 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.860 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:51.709 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.891+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f30381ccca0 con 0x7f301806da70 2024-03-25T02:35:08.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.891+0000 7f302effd640 1 -- 172.21.15.83:0/365897153 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3028009550 con 0x7f301806da70 2024-03-25T02:35:08.881 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 0 to deep-scrub 2024-03-25T02:35:08.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 10 librados: watch_flush enter 2024-03-25T02:35:08.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 10 librados: watch_flush exit 2024-03-25T02:35:08.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 20 client.15375.objecter shutdown clearing up homeless session... 2024-03-25T02:35:08.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 10 client.15375.objecter successfully canceled tick 2024-03-25T02:35:08.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f301806da70 msgr2=0x7f301806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.884 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f301806da70 0x7f301806ff30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3028006fd0 tx=0x7f3028009060 comp rx=0 tx=0).stop 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 10 monclient: shutdown 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 msgr2=0x7f30381cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f30381cc4c0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f302000c400 tx=0x7f302000c8c0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 shutdown_connections 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f301806da70 0x7f301806ff30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f30380b33f0 0x7f3038145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30380a9f50 0x7f30381cc4c0 unknown :-1 s=CLOSED pgs=3199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 --2- 172.21.15.83:0/365897153 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f30380a9b70 0x7f3038142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.894+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 >> 172.21.15.83:0/365897153 conn(0x7f3038067e30 msgr2=0x7f30381400f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.895+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 shutdown_connections 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.895+0000 7f303cd39640 1 -- 172.21.15.83:0/365897153 wait complete. 2024-03-25T02:35:08.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:08.895+0000 7f303cd39640 1 librados: shutdown 2024-03-25T02:35:08.900 INFO:tasks.ceph:Scrubbing osd.1 2024-03-25T02:35:08.900 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:08.907 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:51.712 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.910 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:51.716 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.916 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:51.765 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.920 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:51.769 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.920 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:51.768 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.961 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:51.767 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:08.974 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:51.822 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.023 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.034+0000 7f2e46884640 10 monclient: get_monmap_and_config 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.034+0000 7f2e46884640 10 monclient: build_initial_monmap 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.034+0000 7f2e46884640 10 monclient: monmap: 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:09.036240+0000 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:09.036240+0000 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:09.024 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:09.025 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.035+0000 7f2e46884640 1 Processor -- start 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.035+0000 7f2e46884640 1 -- start start 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.035+0000 7f2e46884640 10 monclient: init 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient(hunting): picked mon.noname-b con 0x7f2e400a9c10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e401403b0 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:09.026 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e40143c90 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:09.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e400abe60 con 0x7f2e40143c90 2024-03-25T02:35:09.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e400ac140 con 0x7f2e401403b0 2024-03-25T02:35:09.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2e401430f0 con 0x7f2e400a9c10 2024-03-25T02:35:09.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:09.027 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.036+0000 7f2e46884640 10 monclient(hunting): authenticate will time out at 3428.226203s 2024-03-25T02:35:09.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.116+0000 7f2e3f7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2e401403b0 0x7f2e40143580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33098/0 (socket says 172.21.15.83:33098) 2024-03-25T02:35:09.105 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.116+0000 7f2e3f7fe640 1 -- 172.21.15.83:0/1225117312 learned_addr learned my addr 172.21.15.83:0/1225117312 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:09.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.147+0000 7f2e3ffff640 1 --2- 172.21.15.83:0/1225117312 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.148+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request con 0x7f2e400a9c10 auth_method 0 2024-03-25T02:35:09.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.148+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:09.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.148+0000 7f2e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:09.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.148+0000 7f2e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:09.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.244+0000 7f2e3effd640 1 -- 172.21.15.83:0/1225117312 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1384493102 0 0) 0x7f2e400abe60 con 0x7f2e40143c90 2024-03-25T02:35:09.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.244+0000 7f2e3effd640 10 monclient(hunting): _negotiate 2024-03-25T02:35:09.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.244+0000 7f2e3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:09.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.244+0000 7f2e3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:09.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.244+0000 7f2e3effd640 10 monclient(hunting): my global_id is 15378 2024-03-25T02:35:09.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.244+0000 7f2e3effd640 1 -- 172.21.15.83:0/1225117312 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e28003520 con 0x7f2e40143c90 2024-03-25T02:35:09.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.267+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:09.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.267+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:09.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.267+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:09.259 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_done global_id 17231 payload 274 2024-03-25T02:35:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 1 -- 172.21.15.83:0/1225117312 >> v1:172.21.15.153:6789/0 conn(0x7f2e401403b0 legacy=0x7f2e40143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 1 -- 172.21.15.83:0/1225117312 >> v1:172.21.15.83:6789/0 conn(0x7f2e40143c90 legacy=0x7f2e4014d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 1 monclient: found mon.noname-b 2024-03-25T02:35:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.270+0000 7f2e3ffff640 1 -- 172.21.15.83:0/1225117312 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e400b3460 con 0x7f2e400a9c10 2024-03-25T02:35:09.273 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.284+0000 7f2e3ffff640 1 --2- 172.21.15.83:0/1225117312 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400a9ff0 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f2e300097d0 tx=0x7f2e3002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83b3e12e9e3c3594 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:09.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 1 -- 172.21.15.83:0/1225117312 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30035020 con 0x7f2e400a9c10 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: dump: 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: _finish_auth 0 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 1 -- 172.21.15.83:0/1225117312 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e30031800 con 0x7f2e400a9c10 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 1 -- 172.21.15.83:0/1225117312 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30031bd0 con 0x7f2e400a9c10 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e46884640 5 monclient: authenticate success, global_id 17231 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e46884640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e46884640 4 set_mon_vals no callback set 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.302+0000 7f2e3effd640 10 monclient: dump: 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.293 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.294 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.294 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.294 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.294 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.294 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.294 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.294 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 10 monclient: shutdown 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 1 -- 172.21.15.83:0/1225117312 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 msgr2=0x7f2e400a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 1 --2- 172.21.15.83:0/1225117312 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400a9ff0 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f2e300097d0 tx=0x7f2e3002b8a0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 1 -- 172.21.15.83:0/1225117312 shutdown_connections 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 1 --2- 172.21.15.83:0/1225117312 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400a9ff0 unknown :-1 s=CLOSED pgs=3198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 1 -- 172.21.15.83:0/1225117312 >> 172.21.15.83:0/1225117312 conn(0x7f2e400b8a60 msgr2=0x7f2e400b8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 1 -- 172.21.15.83:0/1225117312 shutdown_connections 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.303+0000 7f2e46884640 1 -- 172.21.15.83:0/1225117312 wait complete. 2024-03-25T02:35:09.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.304+0000 7f2e46884640 10 monclient: build_initial_monmap 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.304+0000 7f2e46884640 10 monclient: monmap: 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:09.298 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.305+0000 7f2e46884640 1 librados: starting msgr at 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.305+0000 7f2e46884640 1 librados: starting objecter 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.305+0000 7f2e46884640 1 Processor -- start 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.305+0000 7f2e46884640 1 -- start start 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.305+0000 7f2e46884640 1 librados: setting wanted keys 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.305+0000 7f2e46884640 1 librados: calling monclient init 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.305+0000 7f2e46884640 10 monclient: init 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): picked mon.noname-c con 0x7f2e400a9c10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e401403b0 0x7f2e400adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): picked mon.noname-a con 0x7f2e401403b0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40143c90 0x7f2e401d8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): picked mon.noname-b con 0x7f2e40143c90 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2e40140990 con 0x7f2e401403b0 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2e40140810 con 0x7f2e40143c90 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2e40140b10 con 0x7f2e400a9c10 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:09.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.306+0000 7f2e46884640 10 monclient(hunting): authenticate will time out at 3428.496489s 2024-03-25T02:35:09.320 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.331+0000 7f2e3ffff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.321 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.332+0000 7f2e3ffff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50612/0 (socket says 172.21.15.83:50612) 2024-03-25T02:35:09.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.332+0000 7f2e3ffff640 1 -- 172.21.15.83:0/939747141 learned_addr learned my addr 172.21.15.83:0/939747141 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:09.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.332+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request con 0x7f2e400a9c10 auth_method 0 2024-03-25T02:35:09.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.332+0000 7f2e3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:09.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.332+0000 7f2e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:09.322 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.332+0000 7f2e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:09.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.334+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:09.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.334+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:09.323 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.334+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:09.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 10 monclient(hunting): handle_auth_done global_id 17234 payload 824 2024-03-25T02:35:09.327 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 1 -- 172.21.15.83:0/939747141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40143c90 msgr2=0x7f2e401d8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2e40143c90 0x7f2e401d8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 1 -- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e401403b0 msgr2=0x7f2e400adba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2e401403b0 0x7f2e400adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 1 monclient: found mon.noname-c 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.328 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.338+0000 7f2e3ffff640 1 -- 172.21.15.83:0/939747141 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e400af3c0 con 0x7f2e400a9c10 2024-03-25T02:35:09.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.342+0000 7f2e3ffff640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400acfd0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f2e300098f0 tx=0x7f2e300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:09.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.342+0000 7f2e3cff9640 10 client.?.objecter ms_handle_connect 0x7f2e400a9c10 2024-03-25T02:35:09.331 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.342+0000 7f2e3cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:09.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30041070 con 0x7f2e400a9c10 2024-03-25T02:35:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e3cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e3cff9640 10 monclient: dump: 2024-03-25T02:35:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.442 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.443 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.444 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.444 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e3cff9640 10 monclient: _finish_auth 0 2024-03-25T02:35:09.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e3cff9640 10 monclient: _check_auth_tickets 2024-03-25T02:35:09.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:09.444 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 5 monclient: authenticate success, global_id 17234 2024-03-25T02:35:09.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 10 monclient: _renew_subs 2024-03-25T02:35:09.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e401d81e0 con 0x7f2e400a9c10 2024-03-25T02:35:09.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 10 client.17234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:09.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 10 monclient: _renew_subs 2024-03-25T02:35:09.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e400ae0e0 con 0x7f2e400a9c10 2024-03-25T02:35:09.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.451+0000 7f2e46884640 1 librados: init done 2024-03-25T02:35:09.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.452+0000 7f2e46884640 10 client.17234.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:09.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.452+0000 7f2e46884640 15 client.17234.objecter _session_command_op_assign -1 1 2024-03-25T02:35:09.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.452+0000 7f2e46884640 20 client.17234.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:09.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.452+0000 7f2e46884640 10 client.17234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:09.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.452+0000 7f2e46884640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.452+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e40076d20 con 0x7f2e400a9c10 2024-03-25T02:35:09.488 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:52.293 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.489 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:52.294 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.497 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:35:52.302 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.511 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:52.359 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.512 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:52.361 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.520 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:35:52.369 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.563+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e30031800 con 0x7f2e400a9c10 2024-03-25T02:35:09.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.563+0000 7f2e3cff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:09.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.563+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30035040 con 0x7f2e400a9c10 2024-03-25T02:35:09.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.563+0000 7f2e3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.563+0000 7f2e3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.563+0000 7f2e3cff9640 10 monclient: dump: 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.563+0000 7f2e3e7fc640 4 set_mon_vals no callback set 2024-03-25T02:35:09.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.564+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f2e3004c060 con 0x7f2e400a9c10 2024-03-25T02:35:09.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.564+0000 7f2e3cff9640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2e2c06da20 0x7f2e2c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.565+0000 7f2e3f7fe640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2e2c06da20 0x7f2e2c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.565+0000 7f2e3f7fe640 10 monclient: get_auth_request con 0x7f2e2c06da20 auth_method 0 2024-03-25T02:35:09.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.565+0000 7f2e3f7fe640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2e2c06da20 0x7f2e2c06fee0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2e40146db0 tx=0x7f2e34007a40 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:09.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.565+0000 7f2e3cff9640 10 client.17234.objecter ms_handle_connect 0x7f2e2c06da20 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f2e30079310 con 0x7f2e400a9c10 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 10 client.17234.objecter ms_dispatch 0x7f2e40000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 3 client.17234.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 3 client.17234.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 10 client.17234.objecter checking command 1 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f2e2c075780 0x7f2e2c077c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 20 client.17234.objecter _get_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 20 client.17234.objecter put_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 15 client.17234.objecter _session_command_op_remove -1 1 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 20 client.17234.objecter _get_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 20 client.17234.objecter get_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 15 client.17234.objecter _session_command_op_assign 1 1 2024-03-25T02:35:09.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 20 client.17234.objecter put_session s=0x7f2e2c0750b0 osd=1 4 2024-03-25T02:35:09.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 10 client.17234.objecter _send_command 1 2024-03-25T02:35:09.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 --> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e3000af30 con 0x7f2e2c075780 2024-03-25T02:35:09.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.568+0000 7f2e3cff9640 20 client.17234.objecter dump_active .. 0 homeless 2024-03-25T02:35:09.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.569+0000 7f2e44e21640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f2e2c075780 0x7f2e2c077c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.569+0000 7f2e44e21640 10 monclient: get_auth_request con 0x7f2e2c075780 auth_method 0 2024-03-25T02:35:09.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.572+0000 7f2e3cff9640 10 client.17234.objecter ms_handle_connect 0x7f2e2c075780 2024-03-25T02:35:09.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.572+0000 7f2e44e21640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f2e2c075780 0x7f2e2c077c20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:09.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.605+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 <== osd.1 v2:172.21.15.83:6824/2597989034 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e3000af30 con 0x7f2e2c075780 2024-03-25T02:35:09.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.605+0000 7f2e3cff9640 10 client.17234.objecter ms_dispatch 0x7f2e40000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:09.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.605+0000 7f2e3cff9640 10 client.17234.objecter _finish_command 1 = system:0 2024-03-25T02:35:09.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.605+0000 7f2e3cff9640 20 client.17234.objecter put_session s=0x7f2e2c0750b0 osd=1 4 2024-03-25T02:35:09.594 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.605+0000 7f2e3cff9640 15 client.17234.objecter _session_command_op_remove 1 1 2024-03-25T02:35:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 10 client.17234.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 15 client.17234.objecter _session_command_op_assign -1 2 2024-03-25T02:35:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 20 client.17234.objecter _get_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 20 client.17234.objecter put_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.607 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 20 client.17234.objecter _get_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 15 client.17234.objecter _session_command_op_remove -1 2 2024-03-25T02:35:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 20 client.17234.objecter get_session s=0x7f2e2c0750b0 osd=1 3 2024-03-25T02:35:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 15 client.17234.objecter _session_command_op_assign 1 2 2024-03-25T02:35:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 20 client.17234.objecter put_session s=0x7f2e2c0750b0 osd=1 4 2024-03-25T02:35:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 10 client.17234.objecter _send_command 2 2024-03-25T02:35:09.608 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.618+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 --> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2e40075810 con 0x7f2e2c075780 2024-03-25T02:35:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e3cff9640 1 -- 172.21.15.83:0/939747141 <== osd.1 v2:172.21.15.83:6824/2597989034 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2e40075810 con 0x7f2e2c075780 2024-03-25T02:35:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e3cff9640 10 client.17234.objecter ms_dispatch 0x7f2e40000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e3cff9640 10 client.17234.objecter _finish_command 2 = system:0 2024-03-25T02:35:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e3cff9640 20 client.17234.objecter put_session s=0x7f2e2c0750b0 osd=1 4 2024-03-25T02:35:09.627 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e3cff9640 15 client.17234.objecter _session_command_op_remove 1 2 2024-03-25T02:35:09.628 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:09.628 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:09.628 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:09.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e46884640 10 librados: watch_flush enter 2024-03-25T02:35:09.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e46884640 10 librados: watch_flush exit 2024-03-25T02:35:09.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e46884640 10 client.17234.objecter close_session for osd.1 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f2e2c075780 msgr2=0x7f2e2c077c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e46884640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:6824/2597989034,v1:172.21.15.83:6825/2597989034] conn(0x7f2e2c075780 0x7f2e2c077c20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e46884640 20 client.17234.objecter put_session s=0x7f2e2c0750b0 osd=1 1 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.638+0000 7f2e46884640 20 client.17234.objecter shutdown clearing up homeless session... 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 10 client.17234.objecter successfully canceled tick 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2e2c06da20 msgr2=0x7f2e2c06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f2e2c06da20 0x7f2e2c06fee0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2e40146db0 tx=0x7f2e34007a40 comp rx=0 tx=0).stop 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 10 monclient: shutdown 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 20 monclient: shutdown canceling and discarding version request 1 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 msgr2=0x7f2e400acfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 --2- 172.21.15.83:0/939747141 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2e400a9c10 0x7f2e400acfd0 secure :-1 s=READY pgs=3199 cs=0 l=1 rev1=1 crypto rx=0x7f2e300098f0 tx=0x7f2e300046c0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e44e21640 1 -- 172.21.15.83:0/939747141 reap_dead start 2024-03-25T02:35:09.630 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 shutdown_connections 2024-03-25T02:35:09.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 >> 172.21.15.83:0/939747141 conn(0x7f2e400b8a60 msgr2=0x7f2e400b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:09.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 shutdown_connections 2024-03-25T02:35:09.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 -- 172.21.15.83:0/939747141 wait complete. 2024-03-25T02:35:09.631 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.639+0000 7f2e46884640 1 librados: shutdown 2024-03-25T02:35:09.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-25T02:35:09.642 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:52.491 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.644 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:52.492 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.652 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:35:52.501 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:09.745 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.756+0000 7f1259c75640 10 monclient: get_monmap_and_config 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.756+0000 7f1259c75640 10 monclient: build_initial_monmap 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.756+0000 7f1259c75640 10 monclient: monmap: 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:09.757601+0000 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:09.757601+0000 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:09.746 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.756+0000 7f1259c75640 1 Processor -- start 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 1 -- start start 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient: init 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12540a9b70 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12540b33f0 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f12540a9f50 0x7f125414ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient(hunting): picked mon.noname-a con 0x7f12540a9f50 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12540abae0 con 0x7f12540a9b70 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12540abdc0 con 0x7f12540b33f0 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f12540ac0a0 con 0x7f12540a9f50 2024-03-25T02:35:09.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:09.749 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.757+0000 7f1259c75640 10 monclient(hunting): authenticate will time out at 3428.947590s 2024-03-25T02:35:09.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.770+0000 7f1253fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f12540a9f50 0x7f125414ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.777+0000 7f1252ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f12540b33f0 0x7f1254140320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:33110/0 (socket says 172.21.15.83:33110) 2024-03-25T02:35:09.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.777+0000 7f1252ffd640 1 -- 172.21.15.83:0/2763999232 learned_addr learned my addr 172.21.15.83:0/2763999232 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.780+0000 7f1253fff640 10 monclient(hunting): get_auth_request con 0x7f12540a9f50 auth_method 0 2024-03-25T02:35:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.780+0000 7f1253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.780+0000 7f1253fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:09.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.780+0000 7f1253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.782+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.782+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:09.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.782+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:09.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 10 monclient(hunting): handle_auth_done global_id 15381 payload 274 2024-03-25T02:35:09.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 1 -- 172.21.15.83:0/2763999232 >> v1:172.21.15.153:6789/0 conn(0x7f12540b33f0 legacy=0x7f1254140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 1 -- 172.21.15.83:0/2763999232 >> v1:172.21.15.83:6789/0 conn(0x7f12540a9b70 legacy=0x7f12541437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:09.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:09.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 1 monclient: found mon.noname-a 2024-03-25T02:35:09.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:09.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:09.795 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.805+0000 7f1253fff640 1 -- 172.21.15.83:0/2763999232 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12541432a0 con 0x7f12540a9f50 2024-03-25T02:35:09.839 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.850+0000 7f1253fff640 1 --2- 172.21.15.83:0/2763999232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9f50 0x7f125414ea20 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f124c009d90 tx=0x7f124c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec26f2b031aea79 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:09.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 1 -- 172.21.15.83:0/2763999232 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c039070 con 0x7f12540a9f50 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: dump: 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: _finish_auth 0 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: _check_auth_tickets 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 1 -- 172.21.15.83:0/2763999232 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f124c02cc60 con 0x7f12540a9f50 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f1259c75640 5 monclient: authenticate success, global_id 15381 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f1259c75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f1259c75640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 1 -- 172.21.15.83:0/2763999232 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c034660 con 0x7f12540a9f50 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f1259c75640 4 set_mon_vals no callback set 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.881+0000 7f12527fc640 10 monclient: dump: 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.872 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.873 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 10 monclient: shutdown 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 1 -- 172.21.15.83:0/2763999232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9f50 msgr2=0x7f125414ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 1 --2- 172.21.15.83:0/2763999232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9f50 0x7f125414ea20 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f124c009d90 tx=0x7f124c02beb0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 1 -- 172.21.15.83:0/2763999232 shutdown_connections 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 1 --2- 172.21.15.83:0/2763999232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9f50 0x7f125414ea20 unknown :-1 s=CLOSED pgs=3202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 1 -- 172.21.15.83:0/2763999232 >> 172.21.15.83:0/2763999232 conn(0x7f1254067e30 msgr2=0x7f12540b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 1 -- 172.21.15.83:0/2763999232 shutdown_connections 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.882+0000 7f1259c75640 1 -- 172.21.15.83:0/2763999232 wait complete. 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 10 monclient: build_initial_monmap 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 10 monclient: monmap: 2024-03-25T02:35:09.877 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 1 librados: starting msgr at 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 1 librados: starting objecter 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 1 Processor -- start 2024-03-25T02:35:09.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 1 -- start start 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 1 librados: setting wanted keys 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 1 librados: calling monclient init 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.883+0000 7f1259c75640 10 monclient: init 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9b70 0x7f12541d8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): picked mon.noname-a con 0x7f12540a9b70 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12540a9f50 0x7f12540b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): picked mon.noname-b con 0x7f12540a9f50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:09.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f12540b33f0 0x7f12541cff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): picked mon.noname-c con 0x7f12540b33f0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1254140bb0 con 0x7f12540a9b70 2024-03-25T02:35:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1254140a30 con 0x7f12540a9f50 2024-03-25T02:35:09.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f1254140d30 con 0x7f12540b33f0 2024-03-25T02:35:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.884+0000 7f1259c75640 10 monclient(hunting): authenticate will time out at 3429.074259s 2024-03-25T02:35:09.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.889+0000 7f1253fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f12540b33f0 0x7f12541cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.883 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.894+0000 7f12537fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9b70 0x7f12541d8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.896+0000 7f1253fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f12540b33f0 0x7f12541cff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50624/0 (socket says 172.21.15.83:50624) 2024-03-25T02:35:09.885 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.896+0000 7f1253fff640 1 -- 172.21.15.83:0/3620939134 learned_addr learned my addr 172.21.15.83:0/3620939134 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:09.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.896+0000 7f1253fff640 10 monclient(hunting): get_auth_request con 0x7f12540b33f0 auth_method 0 2024-03-25T02:35:09.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.896+0000 7f1253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:09.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.896+0000 7f1253fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:09.886 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.896+0000 7f1253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:09.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.898+0000 7f12537fe640 10 monclient(hunting): get_auth_request con 0x7f12540a9b70 auth_method 0 2024-03-25T02:35:09.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.898+0000 7f12537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:09.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.898+0000 7f12537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:09.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.898+0000 7f12537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:09.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.900+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:09.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.900+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:09.889 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.900+0000 7f1253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.901+0000 7f1252ffd640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12540a9f50 0x7f12540b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.902+0000 7f12537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.902+0000 7f12537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.902+0000 7f12537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.902+0000 7f1252ffd640 10 monclient(hunting): get_auth_request con 0x7f12540a9f50 auth_method 0 2024-03-25T02:35:09.891 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.902+0000 7f1252ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:09.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.902+0000 7f1252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:09.893 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.902+0000 7f1252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.904+0000 7f1252ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.904+0000 7f1252ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.904+0000 7f1252ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 10 monclient(hunting): handle_auth_done global_id 17237 payload 824 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 1 -- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12540a9f50 msgr2=0x7f12540b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12540a9f50 0x7f12540b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 1 -- 172.21.15.83:0/3620939134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9b70 msgr2=0x7f12541d8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9b70 0x7f12541d8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 1 monclient: found mon.noname-c 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:09.895 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.896 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.905+0000 7f1253fff640 1 -- 172.21.15.83:0/3620939134 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12541d9160 con 0x7f12540b33f0 2024-03-25T02:35:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.912+0000 7f1253fff640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f12540b33f0 0x7f12541cff10 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f124c009d90 tx=0x7f124c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.912+0000 7f123bfff640 10 client.?.objecter ms_handle_connect 0x7f12540b33f0 2024-03-25T02:35:09.901 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.912+0000 7f123bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 1 -- 172.21.15.83:0/3620939134 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c044030 con 0x7f12540b33f0 2024-03-25T02:35:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: dump: 2024-03-25T02:35:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.917 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: _finish_auth 0 2024-03-25T02:35:09.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: _check_auth_tickets 2024-03-25T02:35:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 1 -- 172.21.15.83:0/3620939134 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f124c03f470 con 0x7f12540b33f0 2024-03-25T02:35:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 5 monclient: authenticate success, global_id 17237 2024-03-25T02:35:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 10 monclient: _renew_subs 2024-03-25T02:35:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12541d9680 con 0x7f12540b33f0 2024-03-25T02:35:09.919 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 1 -- 172.21.15.83:0/3620939134 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c039040 con 0x7f12540b33f0 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f123bfff640 10 monclient: dump: 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:09.920 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1251ffb640 4 set_mon_vals no callback set 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 10 client.17237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 10 monclient: _renew_subs 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12541cd0b0 con 0x7f12540b33f0 2024-03-25T02:35:09.921 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.927+0000 7f1259c75640 1 librados: init done 2024-03-25T02:35:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.929+0000 7f1259c75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.929+0000 7f1259c75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.929+0000 7f1259c75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:09.922 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:09.929+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1254075810 con 0x7f12540b33f0 2024-03-25T02:35:10.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.203+0000 7f123bfff640 1 -- 172.21.15.83:0/3620939134 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f124c047070 con 0x7f12540b33f0 2024-03-25T02:35:10.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.203+0000 7f123bfff640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f124006da70 0x7f124006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.203+0000 7f12537fe640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f124006da70 0x7f124006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.203+0000 7f12537fe640 10 monclient: get_auth_request con 0x7f124006da70 auth_method 0 2024-03-25T02:35:10.194 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.204+0000 7f12537fe640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f124006da70 0x7f124006ff30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1244006fd0 tx=0x7f1244009060 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:10.195 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.204+0000 7f123bfff640 10 client.17237.objecter ms_handle_connect 0x7f124006da70 2024-03-25T02:35:10.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 1 -- 172.21.15.83:0/3620939134 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f124c07a2c0 con 0x7f12540b33f0 2024-03-25T02:35:10.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 10 client.17237.objecter ms_dispatch 0x7f1254000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:10.243 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 3 client.17237.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:10.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 3 client.17237.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:10.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 20 client.17237.objecter dump_active .. 0 homeless 2024-03-25T02:35:10.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 1 -- 172.21.15.83:0/3620939134 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f124c04e070 con 0x7f12540b33f0 2024-03-25T02:35:10.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:10.244 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.253+0000 7f123bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:10.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.471+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f1254076d20 con 0x7f124006da70 2024-03-25T02:35:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.472+0000 7f123bfff640 1 -- 172.21.15.83:0/3620939134 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1244009550 con 0x7f124006da70 2024-03-25T02:35:10.461 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 1 to deep-scrub 2024-03-25T02:35:10.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 10 librados: watch_flush enter 2024-03-25T02:35:10.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 10 librados: watch_flush exit 2024-03-25T02:35:10.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 20 client.17237.objecter shutdown clearing up homeless session... 2024-03-25T02:35:10.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 10 client.17237.objecter successfully canceled tick 2024-03-25T02:35:10.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f124006da70 msgr2=0x7f124006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f124006da70 0x7f124006ff30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1244006fd0 tx=0x7f1244009060 comp rx=0 tx=0).stop 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 10 monclient: shutdown 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f12540b33f0 msgr2=0x7f12541cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f12540b33f0 0x7f12541cff10 secure :-1 s=READY pgs=3200 cs=0 l=1 rev1=1 crypto rx=0x7f124c009d90 tx=0x7f124c004a40 comp rx=0 tx=0).stop 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 shutdown_connections 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f124006da70 0x7f124006ff30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f12540b33f0 0x7f12541cff10 unknown :-1 s=CLOSED pgs=3200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12540a9f50 0x7f12540b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 --2- 172.21.15.83:0/3620939134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f12540a9b70 0x7f12541d8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 >> 172.21.15.83:0/3620939134 conn(0x7f1254067e30 msgr2=0x7f125413deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:10.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 shutdown_connections 2024-03-25T02:35:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 -- 172.21.15.83:0/3620939134 wait complete. 2024-03-25T02:35:10.466 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.474+0000 7f1259c75640 1 librados: shutdown 2024-03-25T02:35:10.477 INFO:tasks.ceph:Scrubbing osd.2 2024-03-25T02:35:10.477 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:10.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.636+0000 7fae60cff640 10 monclient: get_monmap_and_config 2024-03-25T02:35:10.625 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.636+0000 7fae60cff640 10 monclient: build_initial_monmap 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.636+0000 7fae60cff640 10 monclient: monmap: 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:10.638068+0000 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:10.638068+0000 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:10.626 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 1 Processor -- start 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 1 -- start start 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 10 monclient: init 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae5c0a9c10 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fae5c143e70 0x7fae5c0a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.637+0000 7fae60cff640 10 monclient(hunting): picked mon.noname-a con 0x7fae5c143e70 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.638+0000 7fae60cff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae5c1402c0 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.638+0000 7fae60cff640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae5c0abb80 con 0x7fae5c0a9c10 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.638+0000 7fae60cff640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae5c0abe60 con 0x7fae5c1402c0 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.638+0000 7fae60cff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.628 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.638+0000 7fae60cff640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fae5c140e20 con 0x7fae5c143e70 2024-03-25T02:35:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.638+0000 7fae60cff640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:10.629 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.638+0000 7fae60cff640 10 monclient(hunting): authenticate will time out at 3429.827998s 2024-03-25T02:35:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.646+0000 7fae53fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fae5c143e70 0x7fae5c0a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.646+0000 7fae53fff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fae5c143e70 0x7fae5c0a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49546/0 (socket says 172.21.15.83:49546) 2024-03-25T02:35:10.635 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.646+0000 7fae53fff640 1 -- 172.21.15.83:0/1655334724 learned_addr learned my addr 172.21.15.83:0/1655334724 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:10.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.646+0000 7fae53fff640 10 monclient(hunting): get_auth_request con 0x7fae5c143e70 auth_method 0 2024-03-25T02:35:10.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.646+0000 7fae53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.646+0000 7fae53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.646+0000 7fae53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.647+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.647+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.636 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.647+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 10 monclient(hunting): handle_auth_done global_id 15387 payload 274 2024-03-25T02:35:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 1 -- 172.21.15.83:0/1655334724 >> v1:172.21.15.153:6789/0 conn(0x7fae5c1402c0 legacy=0x7fae5c140710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 1 -- 172.21.15.83:0/1655334724 >> v1:172.21.15.83:6789/0 conn(0x7fae5c0a9c10 legacy=0x7fae5c143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:10.638 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 1 monclient: found mon.noname-a 2024-03-25T02:35:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:10.639 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.649+0000 7fae53fff640 1 -- 172.21.15.83:0/1655334724 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c0ac140 con 0x7fae5c143e70 2024-03-25T02:35:10.640 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.651+0000 7fae53fff640 1 --2- 172.21.15.83:0/1655334724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c143e70 0x7fae5c0a9ff0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fae440098a0 tx=0x7fae4402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d401e0d83ea5a87 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 1 -- 172.21.15.83:0/1655334724 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4402f840 con 0x7fae5c143e70 2024-03-25T02:35:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: dump: 2024-03-25T02:35:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.656 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: _finish_auth 0 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: _check_auth_tickets 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:10.657 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 1 -- 172.21.15.83:0/1655334724 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4402f9e0 con 0x7fae5c143e70 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 1 -- 172.21.15.83:0/1655334724 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4402fdb0 con 0x7fae5c143e70 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae60cff640 5 monclient: authenticate success, global_id 15387 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae60cff640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae60cff640 4 set_mon_vals no callback set 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.660+0000 7fae59d9b640 10 monclient: dump: 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.658 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 10 monclient: shutdown 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 1 -- 172.21.15.83:0/1655334724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c143e70 msgr2=0x7fae5c0a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 1 --2- 172.21.15.83:0/1655334724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c143e70 0x7fae5c0a9ff0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7fae440098a0 tx=0x7fae4402ba20 comp rx=0 tx=0).stop 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 1 -- 172.21.15.83:0/1655334724 shutdown_connections 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 1 --2- 172.21.15.83:0/1655334724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c143e70 0x7fae5c0a9ff0 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 1 -- 172.21.15.83:0/1655334724 >> 172.21.15.83:0/1655334724 conn(0x7fae5c0b5560 msgr2=0x7fae5c0b5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 1 -- 172.21.15.83:0/1655334724 shutdown_connections 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.661+0000 7fae60cff640 1 -- 172.21.15.83:0/1655334724 wait complete. 2024-03-25T02:35:10.659 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.662+0000 7fae60cff640 10 monclient: build_initial_monmap 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.662+0000 7fae60cff640 10 monclient: monmap: 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.662+0000 7fae60cff640 1 librados: starting msgr at 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.662+0000 7fae60cff640 1 librados: starting objecter 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 Processor -- start 2024-03-25T02:35:10.660 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 -- start start 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 librados: setting wanted keys 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 librados: calling monclient init 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient: init 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5c0a9c10 0x7fae5c144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): picked mon.noname-b con 0x7fae5c0a9c10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fae5c1402c0 0x7fae5c1329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): picked mon.noname-c con 0x7fae5c1402c0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c132f30 0x7fae5c130a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): picked mon.noname-a con 0x7fae5c132f30 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c0b3620 con 0x7fae5c132f30 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.661 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c0b34a0 con 0x7fae5c0a9c10 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c14eac0 con 0x7fae5c1402c0 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.663+0000 7fae60cff640 10 monclient(hunting): authenticate will time out at 3429.853746s 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae53fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fae5c1402c0 0x7fae5c1329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5c0a9c10 0x7fae5c144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c132f30 0x7fae5c130a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5c0a9c10 0x7fae5c144790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:59156/0 (socket says 172.21.15.83:59156) 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5a59c640 1 -- 172.21.15.83:0/1224588037 learned_addr learned my addr 172.21.15.83:0/1224588037 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5a59c640 10 monclient(hunting): get_auth_request con 0x7fae5c0a9c10 auth_method 0 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.662 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae53fff640 10 monclient(hunting): get_auth_request con 0x7fae5c1402c0 auth_method 0 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.664+0000 7fae53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5ad9d640 10 monclient(hunting): get_auth_request con 0x7fae5c132f30 auth_method 0 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.663 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.664 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.665+0000 7fae5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 10 monclient(hunting): handle_auth_done global_id 22042 payload 824 2024-03-25T02:35:10.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 1 -- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fae5c1402c0 msgr2=0x7fae5c1329f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fae5c1402c0 0x7fae5c1329f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 1 -- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c132f30 msgr2=0x7fae5c130a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.665 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fae5c132f30 0x7fae5c130a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 10 monclient: _finish_hunting 0 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 1 monclient: found mon.noname-b 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.667+0000 7fae5a59c640 1 -- 172.21.15.83:0/1224588037 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c145aa0 con 0x7fae5c0a9c10 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.670+0000 7fae5a59c640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5c0a9c10 0x7fae5c144790 secure :-1 s=READY pgs=5535 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c400 tx=0x7fae4c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.670+0000 7fae52ffd640 10 client.?.objecter ms_handle_connect 0x7fae5c0a9c10 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.670+0000 7fae52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c0052d0 con 0x7fae5c0a9c10 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.666 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: dump: 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: _finish_auth 0 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:10.667 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4c005490 con 0x7fae5c0a9c10 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c007260 con 0x7fae5c0a9c10 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae52ffd640 10 monclient: dump: 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.668 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 5 monclient: authenticate success, global_id 22042 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 10 monclient: _renew_subs 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5c145d80 con 0x7fae5c0a9c10 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae5959a640 4 set_mon_vals no callback set 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 10 client.22042.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 10 monclient: _renew_subs 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae5c144cd0 con 0x7fae5c0a9c10 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.673+0000 7fae60cff640 1 librados: init done 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.674+0000 7fae60cff640 10 client.22042.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.674+0000 7fae60cff640 15 client.22042.objecter _session_command_op_assign -1 1 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.674+0000 7fae60cff640 20 client.22042.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.674+0000 7fae60cff640 10 client.22042.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:10.669 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.674+0000 7fae60cff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.670 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.674+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae5c076d20 con 0x7fae5c0a9c10 2024-03-25T02:35:10.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.693+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fae4c007400 con 0x7fae5c0a9c10 2024-03-25T02:35:10.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.694+0000 7fae52ffd640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fae3c06da20 0x7fae3c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.694+0000 7fae53fff640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fae3c06da20 0x7fae3c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.683 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.694+0000 7fae53fff640 10 monclient: get_auth_request con 0x7fae3c06da20 auth_method 0 2024-03-25T02:35:10.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.695+0000 7fae52ffd640 10 client.22042.objecter ms_handle_connect 0x7fae3c06da20 2024-03-25T02:35:10.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.695+0000 7fae53fff640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fae3c06da20 0x7fae3c06fee0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fae44036000 tx=0x7fae44008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:10.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7fae4c057090 con 0x7fae5c0a9c10 2024-03-25T02:35:10.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 10 client.22042.objecter ms_dispatch 0x7fae5c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:10.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 3 client.22042.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:10.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 3 client.22042.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:10.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 10 client.22042.objecter checking command 1 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7fae3c075720 0x7fae3c077bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 20 client.22042.objecter _get_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 20 client.22042.objecter put_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 15 client.22042.objecter _session_command_op_remove -1 1 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 20 client.22042.objecter _get_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 20 client.22042.objecter get_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 15 client.22042.objecter _session_command_op_assign 2 1 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 20 client.22042.objecter put_session s=0x7fae3c075050 osd=2 4 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 10 client.22042.objecter _send_command 1 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 --> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae3c078290 con 0x7fae3c075720 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae52ffd640 20 client.22042.objecter dump_active .. 0 homeless 2024-03-25T02:35:10.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.696+0000 7fae5ad9d640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7fae3c075720 0x7fae3c077bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.697+0000 7fae5ad9d640 10 monclient: get_auth_request con 0x7fae3c075720 auth_method 0 2024-03-25T02:35:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.697+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=1250) ==== 24+0+0 (secure 0 0 0) 0x7fae4c055290 con 0x7fae5c0a9c10 2024-03-25T02:35:10.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.697+0000 7fae52ffd640 10 monclient: handle_get_version_reply finishing 1 version 1250 2024-03-25T02:35:10.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.699+0000 7fae52ffd640 10 client.22042.objecter ms_handle_connect 0x7fae3c075720 2024-03-25T02:35:10.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.699+0000 7fae5ad9d640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7fae3c075720 0x7fae3c077bc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:10.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.724+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== osd.2 v2:172.21.15.83:6816/602002618 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae3c078290 con 0x7fae3c075720 2024-03-25T02:35:10.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.724+0000 7fae52ffd640 10 client.22042.objecter ms_dispatch 0x7fae5c000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:10.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.724+0000 7fae52ffd640 10 client.22042.objecter _finish_command 1 = system:0 2024-03-25T02:35:10.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.724+0000 7fae52ffd640 20 client.22042.objecter put_session s=0x7fae3c075050 osd=2 4 2024-03-25T02:35:10.713 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.724+0000 7fae52ffd640 15 client.22042.objecter _session_command_op_remove 2 1 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 10 client.22042.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 15 client.22042.objecter _session_command_op_assign -1 2 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 20 client.22042.objecter _get_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 20 client.22042.objecter put_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 20 client.22042.objecter _get_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 15 client.22042.objecter _session_command_op_remove -1 2 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 20 client.22042.objecter get_session s=0x7fae3c075050 osd=2 3 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 15 client.22042.objecter _session_command_op_assign 2 2 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 20 client.22042.objecter put_session s=0x7fae3c075050 osd=2 4 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 10 client.22042.objecter _send_command 2 2024-03-25T02:35:10.726 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.736+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 --> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fae5c135120 con 0x7fae3c075720 2024-03-25T02:35:10.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.749+0000 7fae52ffd640 1 -- 172.21.15.83:0/1224588037 <== osd.2 v2:172.21.15.83:6816/602002618 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fae54001b20 con 0x7fae3c075720 2024-03-25T02:35:10.738 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.749+0000 7fae52ffd640 10 client.22042.objecter ms_dispatch 0x7fae5c000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:10.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.749+0000 7fae52ffd640 10 client.22042.objecter _finish_command 2 = system:0 2024-03-25T02:35:10.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.749+0000 7fae52ffd640 20 client.22042.objecter put_session s=0x7fae3c075050 osd=2 4 2024-03-25T02:35:10.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.749+0000 7fae52ffd640 15 client.22042.objecter _session_command_op_remove 2 2 2024-03-25T02:35:10.739 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:10.739 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:10.739 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:10.740 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.749+0000 7fae60cff640 10 librados: watch_flush enter 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 10 librados: watch_flush exit 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 10 client.22042.objecter close_session for osd.2 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7fae3c075720 msgr2=0x7fae3c077bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.83:6816/602002618,v1:172.21.15.83:6817/602002618] conn(0x7fae3c075720 0x7fae3c077bc0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 20 client.22042.objecter put_session s=0x7fae3c075050 osd=2 1 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 20 client.22042.objecter shutdown clearing up homeless session... 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 10 client.22042.objecter successfully canceled tick 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fae3c06da20 msgr2=0x7fae3c06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fae3c06da20 0x7fae3c06fee0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fae44036000 tx=0x7fae44008030 comp rx=0 tx=0).stop 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 10 monclient: shutdown 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5c0a9c10 msgr2=0x7fae5c144790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.741 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 --2- 172.21.15.83:0/1224588037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fae5c0a9c10 0x7fae5c144790 secure :-1 s=READY pgs=5535 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c400 tx=0x7fae4c00c8c0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae5ad9d640 1 -- 172.21.15.83:0/1224588037 reap_dead start 2024-03-25T02:35:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 shutdown_connections 2024-03-25T02:35:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 >> 172.21.15.83:0/1224588037 conn(0x7fae5c0b5560 msgr2=0x7fae5c13d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 shutdown_connections 2024-03-25T02:35:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 -- 172.21.15.83:0/1224588037 wait complete. 2024-03-25T02:35:10.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.750+0000 7fae60cff640 1 librados: shutdown 2024-03-25T02:35:10.750 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.863+0000 7f606319d640 10 monclient: get_monmap_and_config 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.863+0000 7f606319d640 10 monclient: build_initial_monmap 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.863+0000 7f606319d640 10 monclient: monmap: 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:10.864540+0000 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:10.864540+0000 2024-03-25T02:35:10.852 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:10.853 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.853 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:10.853 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:10.853 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:10.853 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:10.853 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:10.853 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.863+0000 7f606319d640 1 Processor -- start 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.863+0000 7f606319d640 1 -- start start 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.863+0000 7f606319d640 10 monclient: init 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c0af400 0x7f605c070d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient(hunting): picked mon.noname-b con 0x7f605c0af400 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f605c13fc80 0x7f605c13cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.855 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient(hunting): picked mon.noname-c con 0x7f605c13fc80 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f605c1400a0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605c072bb0 con 0x7f605c1400a0 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f605c0af7e0 con 0x7f605c13fc80 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f605c072980 con 0x7f605c0af400 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.864+0000 7f606319d640 10 monclient(hunting): authenticate will time out at 3430.054501s 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606199a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f605c13fc80 0x7f605c13cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606219b640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c0af400 0x7f605c070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606299c640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f605c1400a0 0x7f605c13d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54218/0 (socket says 172.21.15.83:54218) 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606299c640 1 -- 172.21.15.83:0/1429637046 learned_addr learned my addr 172.21.15.83:0/1429637046 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606199a640 10 monclient(hunting): get_auth_request con 0x7f605c13fc80 auth_method 0 2024-03-25T02:35:10.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606199a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606199a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606199a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606219b640 10 monclient(hunting): get_auth_request con 0x7f605c0af400 auth_method 0 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606219b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606219b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.865+0000 7f606219b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.866+0000 7f606199a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.866+0000 7f606199a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.866+0000 7f606199a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.866+0000 7f606219b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.866+0000 7f606219b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.866+0000 7f606219b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 10 monclient(hunting): handle_auth_done global_id 22045 payload 274 2024-03-25T02:35:10.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 1 -- 172.21.15.83:0/1429637046 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c0af400 msgr2=0x7f605c070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:10.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 1 --2- 172.21.15.83:0/1429637046 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c0af400 0x7f605c070d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 1 -- 172.21.15.83:0/1429637046 >> v1:172.21.15.83:6789/0 conn(0x7f605c1400a0 legacy=0x7f605c13d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 10 monclient: _finish_hunting 0 2024-03-25T02:35:10.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 1 monclient: found mon.noname-c 2024-03-25T02:35:10.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:10.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.867+0000 7f606199a640 1 -- 172.21.15.83:0/1429637046 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c072e90 con 0x7f605c13fc80 2024-03-25T02:35:10.859 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.869+0000 7f606199a640 1 --2- 172.21.15.83:0/1429637046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13fc80 0x7f605c13cd80 secure :-1 s=READY pgs=5536 cs=0 l=1 rev1=1 crypto rx=0x7f604c0097a0 tx=0x7f604c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2e5ddc736f2ab66 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 1 -- 172.21.15.83:0/1429637046 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c034070 con 0x7f605c13fc80 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: dump: 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.863 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: _finish_auth 0 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: _check_auth_tickets 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 1 -- 172.21.15.83:0/1429637046 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604c030610 con 0x7f605c13fc80 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f606319d640 5 monclient: authenticate success, global_id 22045 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f606319d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f606319d640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f606319d640 4 set_mon_vals no callback set 2024-03-25T02:35:10.864 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 1 -- 172.21.15.83:0/1429637046 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604c0309e0 con 0x7f605c13fc80 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.873+0000 7f6061199640 10 monclient: dump: 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.865 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 10 monclient: shutdown 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 -- 172.21.15.83:0/1429637046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13fc80 msgr2=0x7f605c13cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 --2- 172.21.15.83:0/1429637046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13fc80 0x7f605c13cd80 secure :-1 s=READY pgs=5536 cs=0 l=1 rev1=1 crypto rx=0x7f604c0097a0 tx=0x7f604c02b960 comp rx=0 tx=0).stop 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 -- 172.21.15.83:0/1429637046 shutdown_connections 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 --2- 172.21.15.83:0/1429637046 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c13fc80 0x7f605c13cd80 unknown :-1 s=CLOSED pgs=5536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 --2- 172.21.15.83:0/1429637046 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c0af400 0x7f605c070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.866 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 -- 172.21.15.83:0/1429637046 >> 172.21.15.83:0/1429637046 conn(0x7f605c0b4790 msgr2=0x7f605c0b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:10.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 -- 172.21.15.83:0/1429637046 shutdown_connections 2024-03-25T02:35:10.867 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 1 -- 172.21.15.83:0/1429637046 wait complete. 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 10 monclient: build_initial_monmap 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.874+0000 7f606319d640 10 monclient: monmap: 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.871 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 librados: starting msgr at 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 librados: starting objecter 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 Processor -- start 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 -- start start 2024-03-25T02:35:10.872 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 librados: setting wanted keys 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 librados: calling monclient init 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient: init 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c0af400 0x7f605c0adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): picked mon.noname-b con 0x7f605c0af400 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c13fc80 0x7f605c0aa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): picked mon.noname-c con 0x7f605c13fc80 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f605c1400a0 0x7f605c0aaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): picked mon.noname-a con 0x7f605c1400a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f605c0732b0 con 0x7f605c1400a0 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f605c0708a0 con 0x7f605c0af400 2024-03-25T02:35:10.873 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f605c073430 con 0x7f605c13fc80 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.875+0000 7f606319d640 10 monclient(hunting): authenticate will time out at 3430.065854s 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606299c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f605c1400a0 0x7f605c0aaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606199a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c13fc80 0x7f605c0aa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606219b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c0af400 0x7f605c0adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606299c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f605c1400a0 0x7f605c0aaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49568/0 (socket says 172.21.15.83:49568) 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606299c640 1 -- 172.21.15.83:0/2070899324 learned_addr learned my addr 172.21.15.83:0/2070899324 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606299c640 10 monclient(hunting): get_auth_request con 0x7f605c1400a0 auth_method 0 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606299c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606299c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606299c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606199a640 10 monclient(hunting): get_auth_request con 0x7f605c13fc80 auth_method 0 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606199a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606199a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606199a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606219b640 10 monclient(hunting): get_auth_request con 0x7f605c0af400 auth_method 0 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606219b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606219b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.876+0000 7f606219b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606299c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606299c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606299c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606199a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606199a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606199a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.875 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606219b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606219b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.877+0000 7f606219b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 10 monclient(hunting): handle_auth_done global_id 22048 payload 824 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 1 -- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c13fc80 msgr2=0x7f605c0aa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c13fc80 0x7f605c0aa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 1 -- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f605c1400a0 msgr2=0x7f605c0aaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f605c1400a0 0x7f605c0aaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 10 monclient: _finish_hunting 0 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 1 monclient: found mon.noname-b 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606219b640 1 -- 172.21.15.83:0/2070899324 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605c0ac1c0 con 0x7f605c0af400 2024-03-25T02:35:10.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606299c640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f605c1400a0 0x7f605c0aaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T02:35:10.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.879+0000 7f606199a640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c13fc80 0x7f605c0aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T02:35:10.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.882+0000 7f606219b640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c0af400 0x7f605c0adbd0 secure :-1 s=READY pgs=5537 cs=0 l=1 rev1=1 crypto rx=0x7f605800e850 tx=0x7f605800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:10.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.882+0000 7f6052ffd640 10 client.?.objecter ms_handle_connect 0x7f605c0af400 2024-03-25T02:35:10.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.882+0000 7f6052ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 1 -- 172.21.15.83:0/2070899324 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6058017070 con 0x7f605c0af400 2024-03-25T02:35:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.878 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: dump: 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: _finish_auth 0 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:10.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 1 -- 172.21.15.83:0/2070899324 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605800c940 con 0x7f605c0af400 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 1 -- 172.21.15.83:0/2070899324 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605800cd10 con 0x7f605c0af400 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 5 monclient: authenticate success, global_id 22048 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 10 monclient: _renew_subs 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6060998640 4 set_mon_vals no callback set 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605c0ab100 con 0x7f605c0af400 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f6052ffd640 10 monclient: dump: 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:10.880 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 10 client.22048.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 10 monclient: _renew_subs 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605c0ae110 con 0x7f605c0af400 2024-03-25T02:35:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.885+0000 7f606319d640 1 librados: init done 2024-03-25T02:35:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.886+0000 7f606319d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.886+0000 7f606319d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.886+0000 7f606319d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:10.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:10.886+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605c0ae400 con 0x7f605c0af400 2024-03-25T02:35:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.138+0000 7f6052ffd640 1 -- 172.21.15.83:0/2070899324 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f605801a070 con 0x7f605c0af400 2024-03-25T02:35:11.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.138+0000 7f6052ffd640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f604006da70 0x7f604006ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.139+0000 7f606199a640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f604006da70 0x7f604006ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.139+0000 7f606199a640 10 monclient: get_auth_request con 0x7f604006da70 auth_method 0 2024-03-25T02:35:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.139+0000 7f606199a640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f604006da70 0x7f604006ff30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f604c036000 tx=0x7f604c03b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.139+0000 7f6052ffd640 10 client.22048.objecter ms_handle_connect 0x7f604006da70 2024-03-25T02:35:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.141+0000 7f6052ffd640 1 -- 172.21.15.83:0/2070899324 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f6058057a60 con 0x7f605c0af400 2024-03-25T02:35:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.141+0000 7f6052ffd640 10 client.22048.objecter ms_dispatch 0x7f605c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.141+0000 7f6052ffd640 3 client.22048.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.141+0000 7f6052ffd640 3 client.22048.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:11.130 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.141+0000 7f6052ffd640 20 client.22048.objecter dump_active .. 0 homeless 2024-03-25T02:35:11.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.152+0000 7f6052ffd640 1 -- 172.21.15.83:0/2070899324 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f60580599c0 con 0x7f605c0af400 2024-03-25T02:35:11.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.152+0000 7f6052ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:11.140 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.152+0000 7f6052ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:11.314 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.325+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f605c04f940 con 0x7f604006da70 2024-03-25T02:35:11.315 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.326+0000 7f6052ffd640 1 -- 172.21.15.83:0/2070899324 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f605c04f940 con 0x7f604006da70 2024-03-25T02:35:11.315 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 2 to deep-scrub 2024-03-25T02:35:11.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 10 librados: watch_flush enter 2024-03-25T02:35:11.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 10 librados: watch_flush exit 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 20 client.22048.objecter shutdown clearing up homeless session... 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 10 client.22048.objecter successfully canceled tick 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f604006da70 msgr2=0x7f604006ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f604006da70 0x7f604006ff30 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f604c036000 tx=0x7f604c03b040 comp rx=0 tx=0).stop 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 10 monclient: shutdown 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c0af400 msgr2=0x7f605c0adbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c0af400 0x7f605c0adbd0 secure :-1 s=READY pgs=5537 cs=0 l=1 rev1=1 crypto rx=0x7f605800e850 tx=0x7f605800ed10 comp rx=0 tx=0).stop 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 shutdown_connections 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f604006da70 0x7f604006ff30 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f605c1400a0 0x7f605c0aaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f605c13fc80 0x7f605c0aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.318 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 --2- 172.21.15.83:0/2070899324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f605c0af400 0x7f605c0adbd0 unknown :-1 s=CLOSED pgs=5537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 >> 172.21.15.83:0/2070899324 conn(0x7f605c0b4790 msgr2=0x7f605c0afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:11.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 shutdown_connections 2024-03-25T02:35:11.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 -- 172.21.15.83:0/2070899324 wait complete. 2024-03-25T02:35:11.319 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.328+0000 7f606319d640 1 librados: shutdown 2024-03-25T02:35:11.328 INFO:tasks.ceph:Scrubbing osd.3 2024-03-25T02:35:11.328 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:11.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.445+0000 7f55dc2fb640 10 monclient: get_monmap_and_config 2024-03-25T02:35:11.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.445+0000 7f55dc2fb640 10 monclient: build_initial_monmap 2024-03-25T02:35:11.434 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.445+0000 7f55dc2fb640 10 monclient: monmap: 2024-03-25T02:35:11.434 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:11.434 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:11.446469+0000 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:11.446469+0000 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.445+0000 7f55dc2fb640 1 Processor -- start 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.445+0000 7f55dc2fb640 1 -- start start 2024-03-25T02:35:11.435 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.445+0000 7f55dc2fb640 10 monclient: init 2024-03-25T02:35:11.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:11.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-03-25T02:35:11.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55d40a9c10 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:11.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f55d40a9ff0 0x7f55d4143580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient(hunting): picked mon.noname-c con 0x7f55d40a9ff0 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d4143ac0 0x7f55d414d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient(hunting): picked mon.noname-b con 0x7f55d4143ac0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d40abe60 con 0x7f55d40a9c10 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f55d41430f0 con 0x7f55d40a9ff0 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f55d40ac140 con 0x7f55d4143ac0 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55dc2fb640 10 monclient(hunting): authenticate will time out at 3430.636386s 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.446+0000 7f55da898640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d4143ac0 0x7f55d414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55d9896640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f55d40a9ff0 0x7f55d4143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d4143ac0 0x7f55d414d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50660/0 (socket says 172.21.15.83:50660) 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 1 -- 172.21.15.83:0/1677197090 learned_addr learned my addr 172.21.15.83:0/1677197090 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:11.438 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 10 monclient(hunting): get_auth_request con 0x7f55d4143ac0 auth_method 0 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55da898640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55d9896640 10 monclient(hunting): get_auth_request con 0x7f55d40a9ff0 auth_method 0 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55d9896640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55d9896640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.447+0000 7f55d9896640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.448+0000 7f55d9896640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.448+0000 7f55d9896640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.439 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.448+0000 7f55d9896640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 10 monclient(hunting): handle_auth_done global_id 17249 payload 274 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 1 -- 172.21.15.83:0/1677197090 >> v2:172.21.15.153:3300/0 conn(0x7f55d40a9ff0 msgr2=0x7f55d4143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 1 --2- 172.21.15.83:0/1677197090 >> v2:172.21.15.153:3300/0 conn(0x7f55d40a9ff0 0x7f55d4143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 1 -- 172.21.15.83:0/1677197090 >> v1:172.21.15.153:6789/0 conn(0x7f55d40a9c10 legacy=0x7f55d41403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 10 monclient: _finish_hunting 0 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 1 monclient: found mon.noname-b 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:11.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.449+0000 7f55da898640 1 -- 172.21.15.83:0/1677197090 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d4143f30 con 0x7f55d4143ac0 2024-03-25T02:35:11.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.452+0000 7f55da898640 1 --2- 172.21.15.83:0/1677197090 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d4143ac0 0x7f55d414d970 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009e50 tx=0x7f55cc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aedb220c2dc6bc04 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 1 -- 172.21.15.83:0/1677197090 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc02ca70 con 0x7f55d4143ac0 2024-03-25T02:35:11.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:11.445 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: dump: 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: _finish_auth 0 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: _check_auth_tickets 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:11.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 1 -- 172.21.15.83:0/1677197090 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55cc02cc10 con 0x7f55d4143ac0 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55dc2fb640 5 monclient: authenticate success, global_id 17249 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55dc2fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55dc2fb640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55dc2fb640 4 set_mon_vals no callback set 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 1 -- 172.21.15.83:0/1677197090 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0343f0 con 0x7f55d4143ac0 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.456+0000 7f55d9095640 10 monclient: dump: 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.447 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.448 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.448 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.448 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.448 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 10 monclient: shutdown 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 -- 172.21.15.83:0/1677197090 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d4143ac0 msgr2=0x7f55d414d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 --2- 172.21.15.83:0/1677197090 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d4143ac0 0x7f55d414d970 secure :-1 s=READY pgs=3201 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009e50 tx=0x7f55cc02bed0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 -- 172.21.15.83:0/1677197090 shutdown_connections 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 --2- 172.21.15.83:0/1677197090 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d4143ac0 0x7f55d414d970 unknown :-1 s=CLOSED pgs=3201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 --2- 172.21.15.83:0/1677197090 >> v2:172.21.15.153:3300/0 conn(0x7f55d40a9ff0 0x7f55d4143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 -- 172.21.15.83:0/1677197090 >> 172.21.15.83:0/1677197090 conn(0x7f55d40b8a60 msgr2=0x7f55d40b8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 -- 172.21.15.83:0/1677197090 shutdown_connections 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.457+0000 7f55dc2fb640 1 -- 172.21.15.83:0/1677197090 wait complete. 2024-03-25T02:35:11.451 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 10 monclient: build_initial_monmap 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 10 monclient: monmap: 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 1 librados: starting msgr at 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 1 librados: starting objecter 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 1 Processor -- start 2024-03-25T02:35:11.452 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 1 -- start start 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 1 librados: setting wanted keys 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 1 librados: calling monclient init 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.458+0000 7f55dc2fb640 10 monclient: init 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d40a9c10 0x7f55d4146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): picked mon.noname-c con 0x7f55d40a9c10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55d40a9ff0 0x7f55d41dbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): picked mon.noname-a con 0x7f55d40a9ff0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d4143ac0 0x7f55d4137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): picked mon.noname-b con 0x7f55d4143ac0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55d40aaf80 con 0x7f55d40a9ff0 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.453 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55d40aae00 con 0x7f55d4143ac0 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f55d4140ac0 con 0x7f55d40a9c10 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55dc2fb640 10 monclient(hunting): authenticate will time out at 3430.649139s 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55d9896640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55d40a9ff0 0x7f55d41dbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.459+0000 7f55da097640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d40a9c10 0x7f55d4146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d40a9c10 0x7f55d4146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50668/0 (socket says 172.21.15.83:50668) 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 1 -- 172.21.15.83:0/3760572336 learned_addr learned my addr 172.21.15.83:0/3760572336 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 10 monclient(hunting): get_auth_request con 0x7f55d40a9c10 auth_method 0 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d4143ac0 0x7f55d4137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.454 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55d9896640 10 monclient(hunting): get_auth_request con 0x7f55d40a9ff0 auth_method 0 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55d9896640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55d9896640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55d9896640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 10 monclient(hunting): get_auth_request con 0x7f55d4143ac0 auth_method 0 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da097640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.460+0000 7f55da898640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.461+0000 7f55d9896640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.455 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.461+0000 7f55d9896640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.461+0000 7f55d9896640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.456 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 10 monclient(hunting): handle_auth_done global_id 17252 payload 824 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 1 -- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d4143ac0 msgr2=0x7f55d4137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d4143ac0 0x7f55d4137b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 1 -- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55d40a9ff0 msgr2=0x7f55d41dbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55d40a9ff0 0x7f55d41dbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 10 monclient: _finish_hunting 0 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 1 monclient: found mon.noname-c 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.463+0000 7f55da097640 1 -- 172.21.15.83:0/3760572336 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d40adaa0 con 0x7f55d40a9c10 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.466+0000 7f55da097640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d40a9c10 0x7f55d4146fe0 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f55c400c3f0 tx=0x7f55c400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.466+0000 7f55caffd640 10 client.?.objecter ms_handle_connect 0x7f55d40a9c10 2024-03-25T02:35:11.457 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.466+0000 7f55caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:11.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c40052d0 con 0x7f55d40a9c10 2024-03-25T02:35:11.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:11.459 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: dump: 2024-03-25T02:35:11.459 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.459 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: _finish_auth 0 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c4005490 con 0x7f55d40a9c10 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 5 monclient: authenticate success, global_id 17252 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 10 monclient: _renew_subs 2024-03-25T02:35:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d40ac9a0 con 0x7f55d40a9c10 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c4007260 con 0x7f55d40a9c10 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55caffd640 10 monclient: dump: 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.461 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55d8894640 4 set_mon_vals no callback set 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 10 client.17252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 10 monclient: _renew_subs 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d41dc310 con 0x7f55d40a9c10 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.469+0000 7f55dc2fb640 1 librados: init done 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.470+0000 7f55dc2fb640 10 client.17252.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.470+0000 7f55dc2fb640 15 client.17252.objecter _session_command_op_assign -1 1 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.470+0000 7f55dc2fb640 20 client.17252.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.470+0000 7f55dc2fb640 10 client.17252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.470+0000 7f55dc2fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:11.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.470+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f55d4076d20 con 0x7f55d40a9c10 2024-03-25T02:35:11.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.490+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f55c4007400 con 0x7f55d40a9c10 2024-03-25T02:35:11.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.490+0000 7f55caffd640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f55b806da70 0x7f55b806ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.491+0000 7f55d9896640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f55b806da70 0x7f55b806ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.491+0000 7f55d9896640 10 monclient: get_auth_request con 0x7f55b806da70 auth_method 0 2024-03-25T02:35:11.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.491+0000 7f55d9896640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f55b806da70 0x7f55b806ff30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f55d40b84b0 tx=0x7f55d0009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.491+0000 7f55caffd640 10 client.17252.objecter ms_handle_connect 0x7f55b806da70 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f55c4057290 con 0x7f55d40a9c10 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 10 client.17252.objecter ms_dispatch 0x7f55d4000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 3 client.17252.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 3 client.17252.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 10 client.17252.objecter checking command 1 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f55b8075770 0x7f55b8077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 20 client.17252.objecter _get_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 20 client.17252.objecter put_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 15 client.17252.objecter _session_command_op_remove -1 1 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 20 client.17252.objecter _get_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 20 client.17252.objecter get_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 15 client.17252.objecter _session_command_op_assign 3 1 2024-03-25T02:35:11.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 20 client.17252.objecter put_session s=0x7f55b80750a0 osd=3 4 2024-03-25T02:35:11.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 10 client.17252.objecter _send_command 1 2024-03-25T02:35:11.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 --> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55b80782e0 con 0x7f55b8075770 2024-03-25T02:35:11.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.493+0000 7f55caffd640 20 client.17252.objecter dump_active .. 0 homeless 2024-03-25T02:35:11.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.494+0000 7f55da898640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f55b8075770 0x7f55b8077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.494+0000 7f55da898640 10 monclient: get_auth_request con 0x7f55b8075770 auth_method 0 2024-03-25T02:35:11.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.496+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_get_version_reply(handle=1 version=1250) ==== 24+0+0 (secure 0 0 0) 0x7f55c4055490 con 0x7f55d40a9c10 2024-03-25T02:35:11.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.496+0000 7f55caffd640 10 monclient: handle_get_version_reply finishing 1 version 1250 2024-03-25T02:35:11.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.497+0000 7f55da898640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f55b8075770 0x7f55b8077c10 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.497+0000 7f55caffd640 10 client.17252.objecter ms_handle_connect 0x7f55b8075770 2024-03-25T02:35:11.509 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.520+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== osd.3 v2:172.21.15.83:6808/3432501411 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55b80782e0 con 0x7f55b8075770 2024-03-25T02:35:11.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.520+0000 7f55caffd640 10 client.17252.objecter ms_dispatch 0x7f55d4000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:11.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.520+0000 7f55caffd640 10 client.17252.objecter _finish_command 1 = system:0 2024-03-25T02:35:11.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.520+0000 7f55caffd640 20 client.17252.objecter put_session s=0x7f55b80750a0 osd=3 4 2024-03-25T02:35:11.510 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.520+0000 7f55caffd640 15 client.17252.objecter _session_command_op_remove 3 1 2024-03-25T02:35:11.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 10 client.17252.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:11.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 15 client.17252.objecter _session_command_op_assign -1 2 2024-03-25T02:35:11.522 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 20 client.17252.objecter _get_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 20 client.17252.objecter put_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 20 client.17252.objecter _get_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 15 client.17252.objecter _session_command_op_remove -1 2 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 20 client.17252.objecter get_session s=0x7f55b80750a0 osd=3 3 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 15 client.17252.objecter _session_command_op_assign 3 2 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 20 client.17252.objecter put_session s=0x7f55b80750a0 osd=3 4 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 10 client.17252.objecter _send_command 2 2024-03-25T02:35:11.523 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.533+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 --> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f55d41dc5d0 con 0x7f55b8075770 2024-03-25T02:35:11.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55caffd640 1 -- 172.21.15.83:0/3760572336 <== osd.3 v2:172.21.15.83:6808/3432501411 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f55d41dc5d0 con 0x7f55b8075770 2024-03-25T02:35:11.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55caffd640 10 client.17252.objecter ms_dispatch 0x7f55d4000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:11.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55caffd640 10 client.17252.objecter _finish_command 2 = system:0 2024-03-25T02:35:11.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55caffd640 20 client.17252.objecter put_session s=0x7f55b80750a0 osd=3 4 2024-03-25T02:35:11.538 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55caffd640 15 client.17252.objecter _session_command_op_remove 3 2 2024-03-25T02:35:11.538 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:11.538 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:11.539 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 10 librados: watch_flush enter 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 10 librados: watch_flush exit 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 10 client.17252.objecter close_session for osd.3 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f55b8075770 msgr2=0x7f55b8077c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:6808/3432501411,v1:172.21.15.83:6809/3432501411] conn(0x7f55b8075770 0x7f55b8077c10 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 20 client.17252.objecter put_session s=0x7f55b80750a0 osd=3 1 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 20 client.17252.objecter shutdown clearing up homeless session... 2024-03-25T02:35:11.540 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 10 client.17252.objecter successfully canceled tick 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f55b806da70 msgr2=0x7f55b806ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f55b806da70 0x7f55b806ff30 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f55d40b84b0 tx=0x7f55d0009190 comp rx=0 tx=0).stop 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 10 monclient: shutdown 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d40a9c10 msgr2=0x7f55d4146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 --2- 172.21.15.83:0/3760572336 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f55d40a9c10 0x7f55d4146fe0 secure :-1 s=READY pgs=3202 cs=0 l=1 rev1=1 crypto rx=0x7f55c400c3f0 tx=0x7f55c400c8b0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55da898640 1 -- 172.21.15.83:0/3760572336 reap_dead start 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 shutdown_connections 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 >> 172.21.15.83:0/3760572336 conn(0x7f55d40b8a60 msgr2=0x7f55d413ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 shutdown_connections 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.549+0000 7f55dc2fb640 1 -- 172.21.15.83:0/3760572336 wait complete. 2024-03-25T02:35:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.550+0000 7f55dc2fb640 1 librados: shutdown 2024-03-25T02:35:11.549 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-25T02:35:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.679+0000 7fdae2fc8640 10 monclient: get_monmap_and_config 2024-03-25T02:35:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.679+0000 7fdae2fc8640 10 monclient: build_initial_monmap 2024-03-25T02:35:11.668 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.679+0000 7fdae2fc8640 10 monclient: monmap: 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:11.681010+0000 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:11.681010+0000 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:11.669 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 1 Processor -- start 2024-03-25T02:35:11.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 1 -- start start 2024-03-25T02:35:11.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient: init 2024-03-25T02:35:11.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:11.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-03-25T02:35:11.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdadc0a9b70 0x7fdadc1437c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.671 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient(hunting): picked mon.noname-a con 0x7fdadc0a9b70 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdadc0b33f0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdadc0a9f50 0x7fdadc14da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient(hunting): picked mon.noname-c con 0x7fdadc0a9f50 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdadc0abae0 con 0x7fdadc0b33f0 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fdadc0abdc0 con 0x7fdadc0a9b70 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fdadc0ac0a0 con 0x7fdadc0a9f50 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.680+0000 7fdae2fc8640 10 monclient(hunting): authenticate will time out at 3430.870920s 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.681+0000 7fdae0d64640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fdadc0a9b70 0x7fdadc1437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.681+0000 7fdae1565640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdadc0a9f50 0x7fdadc14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.681+0000 7fdae1565640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdadc0a9f50 0x7fdadc14da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:59176/0 (socket says 172.21.15.83:59176) 2024-03-25T02:35:11.672 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 1 -- 172.21.15.83:0/180925867 learned_addr learned my addr 172.21.15.83:0/180925867 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 10 monclient(hunting): get_auth_request con 0x7fdadc0a9f50 auth_method 0 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae0d64640 10 monclient(hunting): get_auth_request con 0x7fdadc0a9b70 auth_method 0 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae0d64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae0d64640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae0d64640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.673 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.682+0000 7fdae1565640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.683+0000 7fdae0d64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.683+0000 7fdae0d64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.683+0000 7fdae0d64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.683+0000 7fdae1565640 10 monclient(hunting): handle_auth_done global_id 22057 payload 274 2024-03-25T02:35:11.674 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 1 -- 172.21.15.83:0/180925867 >> v2:172.21.15.83:3300/0 conn(0x7fdadc0a9b70 msgr2=0x7fdadc1437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 1 --2- 172.21.15.83:0/180925867 >> v2:172.21.15.83:3300/0 conn(0x7fdadc0a9b70 0x7fdadc1437c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 1 -- 172.21.15.83:0/180925867 >> v1:172.21.15.83:6789/0 conn(0x7fdadc0b33f0 legacy=0x7fdadc140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 10 monclient: _finish_hunting 0 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 1 monclient: found mon.noname-c 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.684+0000 7fdae1565640 1 -- 172.21.15.83:0/180925867 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdadc1432a0 con 0x7fdadc0a9f50 2024-03-25T02:35:11.675 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.685+0000 7fdae1565640 1 --2- 172.21.15.83:0/180925867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9f50 0x7fdadc14da10 secure :-1 s=READY pgs=5539 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0b5490 tx=0x7fdad402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=674c75fd7c7c85fb server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 1 -- 172.21.15.83:0/180925867 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad402cb20 con 0x7fdadc0a9f50 2024-03-25T02:35:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:11.678 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: dump: 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: _finish_auth 0 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: _check_auth_tickets 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:11.679 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 1 -- 172.21.15.83:0/180925867 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad402ccc0 con 0x7fdadc0a9f50 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdae2fc8640 5 monclient: authenticate success, global_id 22057 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdae2fc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdae2fc8640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 1 -- 172.21.15.83:0/180925867 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad4034610 con 0x7fdadc0a9f50 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdae2fc8640 4 set_mon_vals no callback set 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.689+0000 7fdadb7fe640 10 monclient: dump: 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.680 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.681 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.681 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.681 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.681 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 10 monclient: shutdown 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 -- 172.21.15.83:0/180925867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9f50 msgr2=0x7fdadc14da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/180925867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9f50 0x7fdadc14da10 secure :-1 s=READY pgs=5539 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0b5490 tx=0x7fdad402bf80 comp rx=0 tx=0).stop 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 -- 172.21.15.83:0/180925867 shutdown_connections 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/180925867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9f50 0x7fdadc14da10 unknown :-1 s=CLOSED pgs=5539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/180925867 >> v2:172.21.15.83:3300/0 conn(0x7fdadc0a9b70 0x7fdadc1437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 -- 172.21.15.83:0/180925867 >> 172.21.15.83:0/180925867 conn(0x7fdadc067e30 msgr2=0x7fdadc0b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 -- 172.21.15.83:0/180925867 shutdown_connections 2024-03-25T02:35:11.684 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.690+0000 7fdae2fc8640 1 -- 172.21.15.83:0/180925867 wait complete. 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.691+0000 7fdae2fc8640 10 monclient: build_initial_monmap 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.691+0000 7fdae2fc8640 10 monclient: monmap: 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.691+0000 7fdae2fc8640 1 librados: starting msgr at 2024-03-25T02:35:11.685 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.691+0000 7fdae2fc8640 1 librados: starting objecter 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.692+0000 7fdae2fc8640 1 Processor -- start 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.692+0000 7fdae2fc8640 1 -- start start 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.692+0000 7fdae2fc8640 1 librados: setting wanted keys 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.692+0000 7fdae2fc8640 1 librados: calling monclient init 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.692+0000 7fdae2fc8640 10 monclient: init 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9b70 0x7fdadc1cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): picked mon.noname-b con 0x7fdadc0a9b70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdadc0a9f50 0x7fdadc1cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): picked mon.noname-a con 0x7fdadc0a9f50 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fdadc0b33f0 0x7fdadc1d9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): picked mon.noname-c con 0x7fdadc0b33f0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:11.686 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc143d00 con 0x7fdadc0a9f50 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc140cc0 con 0x7fdadc0a9b70 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fdadc143e80 con 0x7fdadc0b33f0 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae2fc8640 10 monclient(hunting): authenticate will time out at 3430.883107s 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdadbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdadc0a9f50 0x7fdadc1cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.693+0000 7fdae0d64640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9b70 0x7fdadc1cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdadc0a9f50 0x7fdadc1cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49598/0 (socket says 172.21.15.83:49598) 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdae0d64640 10 monclient(hunting): get_auth_request con 0x7fdadc0a9b70 auth_method 0 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 1 -- 172.21.15.83:0/2684191611 learned_addr learned my addr 172.21.15.83:0/2684191611 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdae0d64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.687 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdae0d64640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdae0d64640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 10 monclient(hunting): get_auth_request con 0x7fdadc0a9f50 auth_method 0 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdae0d64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdae0d64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdae0d64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:11.688 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.694+0000 7fdadbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.696+0000 7fdae0d64640 10 monclient(hunting): handle_auth_done global_id 22060 payload 824 2024-03-25T02:35:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 1 -- 172.21.15.83:0/2684191611 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fdadc0b33f0 msgr2=0x7fdadc1d9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fdadc0b33f0 0x7fdadc1d9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 1 -- 172.21.15.83:0/2684191611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdadc0a9f50 msgr2=0x7fdadc1cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdadc0a9f50 0x7fdadc1cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:11.689 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 10 monclient: _finish_hunting 0 2024-03-25T02:35:11.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 1 monclient: found mon.noname-b 2024-03-25T02:35:11.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:11.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:11.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.697+0000 7fdae0d64640 1 -- 172.21.15.83:0/2684191611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdadc1ceea0 con 0x7fdadc0a9b70 2024-03-25T02:35:11.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.698+0000 7fdae0d64640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9b70 0x7fdadc1cfe60 secure :-1 s=READY pgs=5540 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e850 tx=0x7fdad000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.698+0000 7fdad97fa640 10 client.?.objecter ms_handle_connect 0x7fdadc0a9b70 2024-03-25T02:35:11.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.699+0000 7fdad97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:11.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 1 -- 172.21.15.83:0/2684191611 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0017070 con 0x7fdadc0a9b70 2024-03-25T02:35:11.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:11.691 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: dump: 2024-03-25T02:35:11.691 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: _finish_auth 0 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: _check_auth_tickets 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 1 -- 172.21.15.83:0/2684191611 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad000c940 con 0x7fdadc0a9b70 2024-03-25T02:35:11.692 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 5 monclient: authenticate success, global_id 22060 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 10 monclient: _renew_subs 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdadc1cd5b0 con 0x7fdadc0a9b70 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 1 -- 172.21.15.83:0/2684191611 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad000cd10 con 0x7fdadc0a9b70 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdad97fa640 10 monclient: dump: 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:11.693 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 10 client.22060.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 10 monclient: _renew_subs 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdadaffd640 4 set_mon_vals no callback set 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdadc1cf6d0 con 0x7fdadc0a9b70 2024-03-25T02:35:11.694 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.702+0000 7fdae2fc8640 1 librados: init done 2024-03-25T02:35:11.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.703+0000 7fdae2fc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:11.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.703+0000 7fdae2fc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:11.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.703+0000 7fdae2fc8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:11.695 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.703+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdadc1d9710 con 0x7fdadc0a9b70 2024-03-25T02:35:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.722+0000 7fdad97fa640 1 -- 172.21.15.83:0/2684191611 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fdad001a070 con 0x7fdadc0a9b70 2024-03-25T02:35:11.711 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.722+0000 7fdad97fa640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fdac006da20 0x7fdac006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:11.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.722+0000 7fdadbfff640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fdac006da20 0x7fdac006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:11.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.722+0000 7fdadbfff640 10 monclient: get_auth_request con 0x7fdac006da20 auth_method 0 2024-03-25T02:35:11.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.723+0000 7fdadbfff640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fdac006da20 0x7fdac006fee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdadc13e550 tx=0x7fdac8009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:11.712 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.723+0000 7fdad97fa640 10 client.22060.objecter ms_handle_connect 0x7fdac006da20 2024-03-25T02:35:11.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.951+0000 7fdad97fa640 1 -- 172.21.15.83:0/2684191611 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7fdad0057a50 con 0x7fdadc0a9b70 2024-03-25T02:35:11.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.951+0000 7fdad97fa640 10 client.22060.objecter ms_dispatch 0x7fdadc000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:11.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.951+0000 7fdad97fa640 3 client.22060.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:11.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.951+0000 7fdad97fa640 3 client.22060.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:11.941 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.951+0000 7fdad97fa640 20 client.22060.objecter dump_active .. 0 homeless 2024-03-25T02:35:11.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.962+0000 7fdad97fa640 1 -- 172.21.15.83:0/2684191611 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdad00599b0 con 0x7fdadc0a9b70 2024-03-25T02:35:11.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.962+0000 7fdad97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:11.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:11.962+0000 7fdad97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:12.127 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.138+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fdadc1d8de0 con 0x7fdac006da20 2024-03-25T02:35:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.139+0000 7fdad97fa640 1 -- 172.21.15.83:0/2684191611 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdac8001440 con 0x7fdac006da20 2024-03-25T02:35:12.128 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 3 to deep-scrub 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 10 librados: watch_flush enter 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 10 librados: watch_flush exit 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 20 client.22060.objecter shutdown clearing up homeless session... 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 10 client.22060.objecter successfully canceled tick 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fdac006da20 msgr2=0x7fdac006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fdac006da20 0x7fdac006fee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdadc13e550 tx=0x7fdac8009040 comp rx=0 tx=0).stop 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 10 monclient: shutdown 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9b70 msgr2=0x7fdadc1cfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9b70 0x7fdadc1cfe60 secure :-1 s=READY pgs=5540 cs=0 l=1 rev1=1 crypto rx=0x7fdad000e850 tx=0x7fdad000ed10 comp rx=0 tx=0).stop 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 shutdown_connections 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fdac006da20 0x7fdac006fee0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.131 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fdadc0b33f0 0x7fdadc1d9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdadc0a9f50 0x7fdadc1cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 --2- 172.21.15.83:0/2684191611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdadc0a9b70 0x7fdadc1cfe60 unknown :-1 s=CLOSED pgs=5540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 >> 172.21.15.83:0/2684191611 conn(0x7fdadc067e30 msgr2=0x7fdadc0b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:12.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 shutdown_connections 2024-03-25T02:35:12.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 -- 172.21.15.83:0/2684191611 wait complete. 2024-03-25T02:35:12.132 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.141+0000 7fdae2fc8640 1 librados: shutdown 2024-03-25T02:35:12.142 INFO:tasks.ceph:Scrubbing osd.4 2024-03-25T02:35:12.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.256+0000 7f9cd2035640 10 monclient: get_monmap_and_config 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.257+0000 7f9cd2035640 10 monclient: build_initial_monmap 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.257+0000 7f9cd2035640 10 monclient: monmap: 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:12.258425+0000 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:12.258425+0000 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:12.246 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:12.247 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:12.247 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:12.247 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.248 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.257+0000 7f9cd2035640 1 Processor -- start 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.257+0000 7f9cd2035640 1 -- start start 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.257+0000 7f9cd2035640 10 monclient: init 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ccc06da80 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9ccc0b21b0 0x7f9ccc0b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient(hunting): picked mon.noname-c con 0x7f9ccc0b21b0 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient(hunting): picked mon.noname-b con 0x7f9ccc0b2b30 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ccc06d010 con 0x7f9ccc06da80 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f9ccc06de60 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f9ccc0b1d00 con 0x7f9ccc0b2b30 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9cd2035640 10 monclient(hunting): authenticate will time out at 3431.448338s 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.258+0000 7f9ccbfff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccaffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f9ccc0b21b0 0x7f9ccc0b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc14daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:50684/0 (socket says 172.21.15.83:50684) 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 1 -- 172.21.15.83:0/3218938748 learned_addr learned my addr 172.21.15.83:0/3218938748 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b2b30 auth_method 0 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b21b0 auth_method 0 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.250 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.259+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.251 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.260+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.260+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.260+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_done global_id 17255 payload 274 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 1 -- 172.21.15.83:0/3218938748 >> v2:172.21.15.153:3300/0 conn(0x7f9ccc0b21b0 msgr2=0x7f9ccc0b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 1 --2- 172.21.15.83:0/3218938748 >> v2:172.21.15.153:3300/0 conn(0x7f9ccc0b21b0 0x7f9ccc0b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 1 -- 172.21.15.83:0/3218938748 >> v1:172.21.15.153:6789/0 conn(0x7f9ccc06da80 legacy=0x7f9ccc140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 1 monclient: found mon.noname-b 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.261+0000 7f9ccbfff640 1 -- 172.21.15.83:0/3218938748 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ccc0b1900 con 0x7f9ccc0b2b30 2024-03-25T02:35:12.252 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.263+0000 7f9ccbfff640 1 --2- 172.21.15.83:0/3218938748 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc14daa0 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009e50 tx=0x7f9cc402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eefc0bd78eee7a11 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:12.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 1 -- 172.21.15.83:0/3218938748 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc402cb00 con 0x7f9ccc0b2b30 2024-03-25T02:35:12.255 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: dump: 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: _finish_auth 0 2024-03-25T02:35:12.256 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 1 -- 172.21.15.83:0/3218938748 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc402cca0 con 0x7f9ccc0b2b30 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cd2035640 5 monclient: authenticate success, global_id 17255 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cd2035640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 1 -- 172.21.15.83:0/3218938748 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4034610 con 0x7f9ccc0b2b30 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cd2035640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cd2035640 4 set_mon_vals no callback set 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.266+0000 7f9cca7fc640 10 monclient: dump: 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.257 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.258 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 10 monclient: shutdown 2024-03-25T02:35:12.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:12.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 -- 172.21.15.83:0/3218938748 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 msgr2=0x7f9ccc14daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 --2- 172.21.15.83:0/3218938748 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc14daa0 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009e50 tx=0x7f9cc402bf60 comp rx=0 tx=0).stop 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 -- 172.21.15.83:0/3218938748 shutdown_connections 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 --2- 172.21.15.83:0/3218938748 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc14daa0 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 --2- 172.21.15.83:0/3218938748 >> v2:172.21.15.153:3300/0 conn(0x7f9ccc0b21b0 0x7f9ccc0b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 -- 172.21.15.83:0/3218938748 >> 172.21.15.83:0/3218938748 conn(0x7f9ccc0b6aa0 msgr2=0x7f9ccc0b6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 -- 172.21.15.83:0/3218938748 shutdown_connections 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.267+0000 7f9cd2035640 1 -- 172.21.15.83:0/3218938748 wait complete. 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 10 monclient: build_initial_monmap 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 10 monclient: monmap: 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:12.262 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 1 librados: starting msgr at 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 1 librados: starting objecter 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 1 Processor -- start 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 1 -- start start 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 1 librados: setting wanted keys 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 1 librados: calling monclient init 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.268+0000 7f9cd2035640 10 monclient: init 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc06da80 0x7f9ccc1dc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.263 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): picked mon.noname-a con 0x7f9ccc06da80 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ccc0b21b0 0x7f9ccc0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): picked mon.noname-b con 0x7f9ccc0b21b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): picked mon.noname-c con 0x7f9ccc0b2b30 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ccc140df0 con 0x7f9ccc06da80 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9ccc140c70 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f9ccc0b3210 con 0x7f9ccc0b2b30 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9cd2035640 10 monclient(hunting): authenticate will time out at 3431.459313s 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9ccbfff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9ccb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc06da80 0x7f9ccc1dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.264 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.269+0000 7f9ccaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ccc0b21b0 0x7f9ccc0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ccc0b21b0 0x7f9ccc0afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:59210/0 (socket says 172.21.15.83:59210) 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 1 -- 172.21.15.83:0/2641710599 learned_addr learned my addr 172.21.15.83:0/2641710599 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b21b0 auth_method 0 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccbfff640 10 monclient(hunting): get_auth_request con 0x7f9ccc0b2b30 auth_method 0 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccb7fe640 10 monclient(hunting): get_auth_request con 0x7f9ccc06da80 auth_method 0 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.265 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.270+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.271+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.266 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.271+0000 7f9ccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 10 monclient(hunting): handle_auth_done global_id 22066 payload 824 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 1 -- 172.21.15.83:0/2641710599 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 msgr2=0x7f9ccc0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f9ccc0b2b30 0x7f9ccc0b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 1 -- 172.21.15.83:0/2641710599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc06da80 msgr2=0x7f9ccc1dc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc06da80 0x7f9ccc1dc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 10 monclient: _finish_hunting 0 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 1 monclient: found mon.noname-b 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:12.267 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.273+0000 7f9ccaffd640 1 -- 172.21.15.83:0/2641710599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ccc0af870 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.275+0000 7f9ccaffd640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ccc0b21b0 0x7f9ccc0afc90 secure :-1 s=READY pgs=5542 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000e9b0 tx=0x7f9cc000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.275+0000 7f9cb3fff640 10 client.?.objecter ms_handle_connect 0x7f9ccc0b21b0 2024-03-25T02:35:12.268 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.275+0000 7f9cb3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0017070 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: dump: 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.269 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: _finish_auth 0 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: _check_auth_tickets 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc000c9a0 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 5 monclient: authenticate success, global_id 22066 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc000cd90 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 monclient: _renew_subs 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:12.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ccc1dc7a0 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cc9ffb640 4 set_mon_vals no callback set 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cb3fff640 10 monclient: dump: 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 client.22066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:12.271 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 monclient: _renew_subs 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ccc0b08f0 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 1 librados: init done 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 client.22066.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 15 client.22066.objecter _session_command_op_assign -1 1 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 20 client.22066.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 client.22066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:12.272 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.279+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ccc076d20 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.299+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f9cc001a070 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.299+0000 7f9cb3fff640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f9cb806da20 0x7f9cb806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.299+0000 7f9ccb7fe640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f9cb806da20 0x7f9cb806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.288 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.299+0000 7f9ccb7fe640 10 monclient: get_auth_request con 0x7f9cb806da20 auth_method 0 2024-03-25T02:35:12.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.300+0000 7f9cb3fff640 10 client.22066.objecter ms_handle_connect 0x7f9cb806da20 2024-03-25T02:35:12.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.300+0000 7f9ccb7fe640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f9cb806da20 0x7f9cb806fee0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc001780 tx=0x7f9cbc009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:12.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f9cc00576c0 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 10 client.22066.objecter ms_dispatch 0x7f9ccc000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 3 client.22066.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 3 client.22066.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 10 client.22066.objecter checking command 1 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9cb8075720 0x7f9cb8077bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 20 client.22066.objecter _get_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 20 client.22066.objecter put_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 15 client.22066.objecter _session_command_op_remove -1 1 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 20 client.22066.objecter _get_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 20 client.22066.objecter get_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 15 client.22066.objecter _session_command_op_assign 4 1 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 20 client.22066.objecter put_session s=0x7f9cb8075050 osd=4 4 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 10 client.22066.objecter _send_command 1 2024-03-25T02:35:12.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 --> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cc0001d30 con 0x7f9cb8075720 2024-03-25T02:35:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9cb3fff640 20 client.22066.objecter dump_active .. 0 homeless 2024-03-25T02:35:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.301+0000 7f9ccbfff640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9cb8075720 0x7f9cb8077bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.302+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=1250) ==== 24+0+0 (secure 0 0 0) 0x7f9cc00558c0 con 0x7f9ccc0b21b0 2024-03-25T02:35:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.302+0000 7f9cb3fff640 10 monclient: handle_get_version_reply finishing 1 version 1250 2024-03-25T02:35:12.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.302+0000 7f9ccbfff640 10 monclient: get_auth_request con 0x7f9cb8075720 auth_method 0 2024-03-25T02:35:12.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.305+0000 7f9ccbfff640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9cb8075720 0x7f9cb8077bc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:12.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.305+0000 7f9cb3fff640 10 client.22066.objecter ms_handle_connect 0x7f9cb8075720 2024-03-25T02:35:12.316 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.327+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== osd.4 v2:172.21.15.153:6818/2579144563 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9cc0001d30 con 0x7f9cb8075720 2024-03-25T02:35:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.328+0000 7f9cb3fff640 10 client.22066.objecter ms_dispatch 0x7f9ccc000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.328+0000 7f9cb3fff640 10 client.22066.objecter _finish_command 1 = system:0 2024-03-25T02:35:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.328+0000 7f9cb3fff640 20 client.22066.objecter put_session s=0x7f9cb8075050 osd=4 4 2024-03-25T02:35:12.317 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.328+0000 7f9cb3fff640 15 client.22066.objecter _session_command_op_remove 4 1 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 10 client.22066.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 15 client.22066.objecter _session_command_op_assign -1 2 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 20 client.22066.objecter _get_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 20 client.22066.objecter put_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 20 client.22066.objecter _get_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 15 client.22066.objecter _session_command_op_remove -1 2 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 20 client.22066.objecter get_session s=0x7f9cb8075050 osd=4 3 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 15 client.22066.objecter _session_command_op_assign 4 2 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 20 client.22066.objecter put_session s=0x7f9cb8075050 osd=4 4 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 10 client.22066.objecter _send_command 2 2024-03-25T02:35:12.330 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.341+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 --> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9ccc06bd50 con 0x7f9cb8075720 2024-03-25T02:35:12.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cb3fff640 1 -- 172.21.15.83:0/2641710599 <== osd.4 v2:172.21.15.153:6818/2579144563 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9ccc06bd50 con 0x7f9cb8075720 2024-03-25T02:35:12.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cb3fff640 10 client.22066.objecter ms_dispatch 0x7f9ccc000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:12.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cb3fff640 10 client.22066.objecter _finish_command 2 = system:0 2024-03-25T02:35:12.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cb3fff640 20 client.22066.objecter put_session s=0x7f9cb8075050 osd=4 4 2024-03-25T02:35:12.346 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cb3fff640 15 client.22066.objecter _session_command_op_remove 4 2 2024-03-25T02:35:12.347 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:12.347 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:12.347 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 10 librados: watch_flush enter 2024-03-25T02:35:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 10 librados: watch_flush exit 2024-03-25T02:35:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 10 client.22066.objecter close_session for osd.4 2024-03-25T02:35:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9cb8075720 msgr2=0x7f9cb8077bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6818/2579144563,v1:172.21.15.153:6819/2579144563] conn(0x7f9cb8075720 0x7f9cb8077bc0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.348 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 20 client.22066.objecter put_session s=0x7f9cb8075050 osd=4 1 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 20 client.22066.objecter shutdown clearing up homeless session... 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 10 client.22066.objecter successfully canceled tick 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f9cb806da20 msgr2=0x7f9cb806fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f9cb806da20 0x7f9cb806fee0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc001780 tx=0x7f9cbc009040 comp rx=0 tx=0).stop 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 10 monclient: shutdown 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ccc0b21b0 msgr2=0x7f9ccc0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 --2- 172.21.15.83:0/2641710599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ccc0b21b0 0x7f9ccc0afc90 secure :-1 s=READY pgs=5542 cs=0 l=1 rev1=1 crypto rx=0x7f9cc000e9b0 tx=0x7f9cc000ee70 comp rx=0 tx=0).stop 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9ccbfff640 1 -- 172.21.15.83:0/2641710599 reap_dead start 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 shutdown_connections 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 >> 172.21.15.83:0/2641710599 conn(0x7f9ccc0b6aa0 msgr2=0x7f9ccc0b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 shutdown_connections 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 -- 172.21.15.83:0/2641710599 wait complete. 2024-03-25T02:35:12.349 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.357+0000 7f9cd2035640 1 librados: shutdown 2024-03-25T02:35:12.357 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-25T02:35:12.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.469+0000 7f92215bf640 10 monclient: get_monmap_and_config 2024-03-25T02:35:12.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.469+0000 7f92215bf640 10 monclient: build_initial_monmap 2024-03-25T02:35:12.458 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.469+0000 7f92215bf640 10 monclient: monmap: 2024-03-25T02:35:12.458 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:12.458 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:12.470853+0000 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:12.470853+0000 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:12.459 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 1 Processor -- start 2024-03-25T02:35:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 1 -- start start 2024-03-25T02:35:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient: init 2024-03-25T02:35:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:12.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c1437c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient(hunting): picked mon.noname-b con 0x7f921c0a9b70 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f921c0b33f0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f921c0a9f50 0x7f921c14da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient(hunting): picked mon.noname-c con 0x7f921c0a9f50 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921c0abae0 con 0x7f921c0b33f0 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f921c0abdc0 con 0x7f921c0a9f50 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f921c0ac0a0 con 0x7f921c0a9b70 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.470+0000 7f92215bf640 10 monclient(hunting): authenticate will time out at 3431.660810s 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f921affd640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c1437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.462 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f9213fff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f921c0b33f0 0x7f921c140320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54230/0 (socket says 172.21.15.83:54230) 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f9213fff640 1 -- 172.21.15.83:0/3424332074 learned_addr learned my addr 172.21.15.83:0/3424332074 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f921b7fe640 1 --2- 172.21.15.83:0/3424332074 >> v2:172.21.15.153:3300/0 conn(0x7f921c0a9f50 0x7f921c14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f921affd640 10 monclient(hunting): get_auth_request con 0x7f921c0a9b70 auth_method 0 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f921affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f921affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.471+0000 7f921affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921b7fe640 10 monclient(hunting): get_auth_request con 0x7f921c0a9f50 auth_method 0 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.463 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.472+0000 7f921b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921a7fc640 1 -- 172.21.15.83:0/3424332074 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998961423 0 0) 0x7f921c0abae0 con 0x7f921c0b33f0 2024-03-25T02:35:12.464 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 10 monclient(hunting): handle_auth_done global_id 17261 payload 274 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 1 -- 172.21.15.83:0/3424332074 >> v2:172.21.15.153:3300/0 conn(0x7f921c0a9f50 msgr2=0x7f921c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 1 --2- 172.21.15.83:0/3424332074 >> v2:172.21.15.153:3300/0 conn(0x7f921c0a9f50 0x7f921c14da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 1 -- 172.21.15.83:0/3424332074 >> v1:172.21.15.83:6789/0 conn(0x7f921c0b33f0 legacy=0x7f921c140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 10 monclient: _finish_hunting 0 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 1 monclient: found mon.noname-b 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921affd640 1 -- 172.21.15.83:0/3424332074 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c1432a0 con 0x7f921c0a9b70 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.474+0000 7f921a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-03-25T02:35:12.465 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.476+0000 7f921affd640 1 --2- 172.21.15.83:0/3424332074 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c1437c0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f920c0097a0 tx=0x7f920c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=901d7f8040c7fc04 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 1 -- 172.21.15.83:0/3424332074 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c035020 con 0x7f921c0a9b70 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: dump: 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.469 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: _finish_auth 0 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: _check_auth_tickets 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 5 monclient: authenticate success, global_id 17261 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 1 -- 172.21.15.83:0/3424332074 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c031830 con 0x7f921c0a9b70 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:12.470 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 4 set_mon_vals no callback set 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 1 -- 172.21.15.83:0/3424332074 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f920c031c00 con 0x7f921c0a9b70 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f921a7fc640 10 monclient: dump: 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.471 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 10 monclient: shutdown 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 -- 172.21.15.83:0/3424332074 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 msgr2=0x7f921c1437c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 --2- 172.21.15.83:0/3424332074 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c1437c0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f920c0097a0 tx=0x7f920c02b8d0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 -- 172.21.15.83:0/3424332074 shutdown_connections 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 --2- 172.21.15.83:0/3424332074 >> v2:172.21.15.153:3300/0 conn(0x7f921c0a9f50 0x7f921c14da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 --2- 172.21.15.83:0/3424332074 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c1437c0 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 -- 172.21.15.83:0/3424332074 >> 172.21.15.83:0/3424332074 conn(0x7f921c067e30 msgr2=0x7f921c0b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:12.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 -- 172.21.15.83:0/3424332074 shutdown_connections 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.480+0000 7f92215bf640 1 -- 172.21.15.83:0/3424332074 wait complete. 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 10 monclient: build_initial_monmap 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 10 monclient: monmap: 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:12.476 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 1 librados: starting msgr at 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 1 librados: starting objecter 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 1 Processor -- start 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 1 -- start start 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 1 librados: setting wanted keys 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 1 librados: calling monclient init 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.481+0000 7f92215bf640 10 monclient: init 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c0b8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): picked mon.noname-c con 0x7f921c0a9b70 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921c0a9f50 0x7f921c1d8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): picked mon.noname-b con 0x7f921c0a9f50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:12.477 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f921c0b33f0 0x7f921c136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): picked mon.noname-a con 0x7f921c0b33f0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f921c143d00 con 0x7f921c0b33f0 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f921c140c40 con 0x7f921c0a9f50 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f921c143e80 con 0x7f921c0a9b70 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.482+0000 7f92215bf640 10 monclient(hunting): authenticate will time out at 3431.672486s 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f921c0b33f0 0x7f921c136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f921c0b33f0 0x7f921c136e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49630/0 (socket says 172.21.15.83:49630) 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 1 -- 172.21.15.83:0/2169586623 learned_addr learned my addr 172.21.15.83:0/2169586623 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:12.478 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 10 monclient(hunting): get_auth_request con 0x7f921c0b33f0 auth_method 0 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921affd640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c0b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.483+0000 7f921b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f9213fff640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921c0a9f50 0x7f921c1d8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921affd640 10 monclient(hunting): get_auth_request con 0x7f921c0a9b70 auth_method 0 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f9213fff640 10 monclient(hunting): get_auth_request con 0x7f921c0a9f50 auth_method 0 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f9213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:12.479 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f9213fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:12.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f9213fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:12.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.484+0000 7f921affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.485+0000 7f9213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:12.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.485+0000 7f9213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:12.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.485+0000 7f9213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 10 monclient(hunting): handle_auth_done global_id 15417 payload 824 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 1 -- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 msgr2=0x7f921c0b8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c0b8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 1 -- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921c0a9f50 msgr2=0x7f921c1d8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921c0a9f50 0x7f921c1d8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 10 monclient: _finish_hunting 0 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 1 monclient: found mon.noname-a 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:12.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.486+0000 7f921b7fe640 1 -- 172.21.15.83:0/2169586623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c0b97f0 con 0x7f921c0b33f0 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.489+0000 7f921b7fe640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f921c0b33f0 0x7f921c136e70 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f921400efc0 tx=0x7f921400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.489+0000 7f92137fe640 10 client.?.objecter ms_handle_connect 0x7f921c0b33f0 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.489+0000 7f92137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 1 -- 172.21.15.83:0/2169586623 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214018020 con 0x7f921c0b33f0 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: dump: 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.484 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: _finish_auth 0 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: _check_auth_tickets 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 1 -- 172.21.15.83:0/2169586623 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9214014480 con 0x7f921c0b33f0 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92215bf640 5 monclient: authenticate success, global_id 15417 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92215bf640 10 monclient: _renew_subs 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92215bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921c0b8860 con 0x7f921c0b33f0 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 1 -- 172.21.15.83:0/2169586623 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214014870 con 0x7f921c0b33f0 2024-03-25T02:35:12.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92137fe640 10 monclient: dump: 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:12.486 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f9219ffb640 4 set_mon_vals no callback set 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.492+0000 7f92215bf640 10 client.15417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 10 monclient: _renew_subs 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f921c0b75e0 con 0x7f921c0b33f0 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 1 librados: init done 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:12.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.493+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921c075810 con 0x7f921c0b33f0 2024-03-25T02:35:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.512+0000 7f92137fe640 1 -- 172.21.15.83:0/2169586623 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f9214005d10 con 0x7f921c0b33f0 2024-03-25T02:35:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.512+0000 7f92137fe640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f91fc06da70 0x7f91fc06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.512+0000 7f921affd640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f91fc06da70 0x7f91fc06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:12.501 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.512+0000 7f921affd640 10 monclient: get_auth_request con 0x7f91fc06da70 auth_method 0 2024-03-25T02:35:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.513+0000 7f921affd640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f91fc06da70 0x7f91fc06ff30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f920c036000 tx=0x7f920c008030 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:12.502 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.513+0000 7f92137fe640 10 client.15417.objecter ms_handle_connect 0x7f91fc06da70 2024-03-25T02:35:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.515+0000 7f92137fe640 1 -- 172.21.15.83:0/2169586623 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f9214058690 con 0x7f921c0b33f0 2024-03-25T02:35:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.515+0000 7f92137fe640 10 client.15417.objecter ms_dispatch 0x7f921c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.515+0000 7f92137fe640 3 client.15417.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.515+0000 7f92137fe640 3 client.15417.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:12.504 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.515+0000 7f92137fe640 20 client.15417.objecter dump_active .. 0 homeless 2024-03-25T02:35:12.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.764+0000 7f92137fe640 1 -- 172.21.15.83:0/2169586623 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9214022030 con 0x7f921c0b33f0 2024-03-25T02:35:12.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.764+0000 7f92137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:12.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.764+0000 7f92137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:12.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.951+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f921c076d20 con 0x7f91fc06da70 2024-03-25T02:35:12.940 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.951+0000 7f92137fe640 1 -- 172.21.15.83:0/2169586623 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f920c031860 con 0x7f91fc06da70 2024-03-25T02:35:12.940 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 4 to deep-scrub 2024-03-25T02:35:12.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 10 librados: watch_flush enter 2024-03-25T02:35:12.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 10 librados: watch_flush exit 2024-03-25T02:35:12.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 20 client.15417.objecter shutdown clearing up homeless session... 2024-03-25T02:35:12.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 10 client.15417.objecter successfully canceled tick 2024-03-25T02:35:12.943 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f91fc06da70 msgr2=0x7f91fc06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f91fc06da70 0x7f91fc06ff30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f920c036000 tx=0x7f920c008030 comp rx=0 tx=0).stop 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 10 monclient: shutdown 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f921c0b33f0 msgr2=0x7f921c136e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f921c0b33f0 0x7f921c136e70 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f921400efc0 tx=0x7f921400c490 comp rx=0 tx=0).stop 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 shutdown_connections 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f91fc06da70 0x7f91fc06ff30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f921c0b33f0 0x7f921c136e70 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f921c0a9f50 0x7f921c1d8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 --2- 172.21.15.83:0/2169586623 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f921c0a9b70 0x7f921c0b8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 >> 172.21.15.83:0/2169586623 conn(0x7f921c067e30 msgr2=0x7f921c0b54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 shutdown_connections 2024-03-25T02:35:12.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 -- 172.21.15.83:0/2169586623 wait complete. 2024-03-25T02:35:12.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:12.953+0000 7f92215bf640 1 librados: shutdown 2024-03-25T02:35:12.954 INFO:tasks.ceph:Scrubbing osd.5 2024-03-25T02:35:12.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:13.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.069+0000 7f9137e84640 10 monclient: get_monmap_and_config 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.069+0000 7f9137e84640 10 monclient: build_initial_monmap 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.069+0000 7f9137e84640 10 monclient: monmap: 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:13.070980+0000 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:13.070980+0000 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 1 Processor -- start 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 1 -- start start 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient: init 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f91300a9c10 0x7f91300a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient(hunting): picked mon.noname-c con 0x7f91300a9c10 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91301403b0 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9130143c90 0x7f9130140810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient(hunting): picked mon.noname-a con 0x7f9130143c90 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91300abe60 con 0x7f91301403b0 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f91301430f0 con 0x7f9130143c90 2024-03-25T02:35:13.062 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f91300ac140 con 0x7f91300a9c10 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.070+0000 7f9137e84640 10 monclient(hunting): authenticate will time out at 3432.260828s 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9136421640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9130143c90 0x7f9130140810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9135c20640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f91300a9c10 0x7f91300a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9136421640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f9130143c90 0x7f9130140810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49642/0 (socket says 172.21.15.83:49642) 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9136421640 1 -- 172.21.15.83:0/516563824 learned_addr learned my addr 172.21.15.83:0/516563824 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9136421640 10 monclient(hunting): get_auth_request con 0x7f9130143c90 auth_method 0 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9136421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9136421640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.071+0000 7f9136421640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9135c20640 10 monclient(hunting): get_auth_request con 0x7f91300a9c10 auth_method 0 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9135c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.063 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9135c20640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9135c20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9136421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9136421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9136421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9135c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9135c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.064 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.072+0000 7f9135c20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.073+0000 7f9136421640 10 monclient(hunting): handle_auth_done global_id 15420 payload 274 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 1 -- 172.21.15.83:0/516563824 >> v2:172.21.15.153:3300/0 conn(0x7f91300a9c10 msgr2=0x7f91300a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 1 --2- 172.21.15.83:0/516563824 >> v2:172.21.15.153:3300/0 conn(0x7f91300a9c10 0x7f91300a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 1 -- 172.21.15.83:0/516563824 >> v1:172.21.15.153:6789/0 conn(0x7f91301403b0 legacy=0x7f9130143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 10 monclient: _finish_hunting 0 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 1 monclient: found mon.noname-a 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:13.065 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.074+0000 7f9136421640 1 -- 172.21.15.83:0/516563824 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91300b3460 con 0x7f9130143c90 2024-03-25T02:35:13.066 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.075+0000 7f9136421640 1 --2- 172.21.15.83:0/516563824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 0x7f9130140810 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f91300b54c0 tx=0x7f912802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0ddfd5918d37f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 1 -- 172.21.15.83:0/516563824 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912802ca80 con 0x7f9130143c90 2024-03-25T02:35:13.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:13.068 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: dump: 2024-03-25T02:35:13.068 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.068 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: _finish_auth 0 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: _check_auth_tickets 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 1 -- 172.21.15.83:0/516563824 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912802cc20 con 0x7f9130143c90 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:13.069 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 1 -- 172.21.15.83:0/516563824 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91280343f0 con 0x7f9130143c90 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9134c1e640 10 monclient: dump: 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.070 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9137e84640 5 monclient: authenticate success, global_id 15420 2024-03-25T02:35:13.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9137e84640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:13.071 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.079+0000 7f9137e84640 4 set_mon_vals no callback set 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 10 monclient: shutdown 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 -- 172.21.15.83:0/516563824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 msgr2=0x7f9130140810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 --2- 172.21.15.83:0/516563824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 0x7f9130140810 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f91300b54c0 tx=0x7f912802bee0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 -- 172.21.15.83:0/516563824 shutdown_connections 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 --2- 172.21.15.83:0/516563824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 0x7f9130140810 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 --2- 172.21.15.83:0/516563824 >> v2:172.21.15.153:3300/0 conn(0x7f91300a9c10 0x7f91300a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 -- 172.21.15.83:0/516563824 >> 172.21.15.83:0/516563824 conn(0x7f91300b8a60 msgr2=0x7f91300b8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:13.074 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 -- 172.21.15.83:0/516563824 shutdown_connections 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.080+0000 7f9137e84640 1 -- 172.21.15.83:0/516563824 wait complete. 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 10 monclient: build_initial_monmap 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 10 monclient: monmap: 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 1 librados: starting msgr at 2024-03-25T02:35:13.075 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 1 librados: starting objecter 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 1 Processor -- start 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 1 -- start start 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 1 librados: setting wanted keys 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 1 librados: calling monclient init 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.081+0000 7f9137e84640 10 monclient: init 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91300a9c10 0x7f91301420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): picked mon.noname-b con 0x7f91300a9c10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f91301403b0 0x7f9130133090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): picked mon.noname-c con 0x7f91301403b0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 0x7f91301348a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): picked mon.noname-a con 0x7f9130143c90 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.076 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f913014daf0 con 0x7f9130143c90 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f913014d970 con 0x7f91300a9c10 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f913014dc70 con 0x7f91301403b0 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9137e84640 10 monclient(hunting): authenticate will time out at 3432.272125s 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f913541f640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f91301403b0 0x7f9130133090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.082+0000 7f9135c20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91300a9c10 0x7f91301420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91300a9c10 0x7f91301420e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:59224/0 (socket says 172.21.15.83:59224) 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 1 -- 172.21.15.83:0/1862664057 learned_addr learned my addr 172.21.15.83:0/1862664057 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 10 monclient(hunting): get_auth_request con 0x7f91300a9c10 auth_method 0 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.077 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f913541f640 10 monclient(hunting): get_auth_request con 0x7f91301403b0 auth_method 0 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f913541f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f913541f640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f913541f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9135c20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f913541f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f913541f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f913541f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.083+0000 7f9136421640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 0x7f91301348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.084+0000 7f9136421640 10 monclient(hunting): get_auth_request con 0x7f9130143c90 auth_method 0 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.084+0000 7f9136421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.084+0000 7f9136421640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.078 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.084+0000 7f9136421640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.084+0000 7f9136421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.084+0000 7f9136421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.079 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.084+0000 7f9136421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 10 monclient(hunting): handle_auth_done global_id 22081 payload 824 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 1 -- 172.21.15.83:0/1862664057 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f91301403b0 msgr2=0x7f9130133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f91301403b0 0x7f9130133090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 1 -- 172.21.15.83:0/1862664057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 msgr2=0x7f91301348a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9130143c90 0x7f91301348a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 10 monclient: _finish_hunting 0 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 1 monclient: found mon.noname-b 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.086+0000 7f9135c20640 1 -- 172.21.15.83:0/1862664057 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9130142ec0 con 0x7f91300a9c10 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.088+0000 7f9135c20640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91300a9c10 0x7f91301420e0 secure :-1 s=READY pgs=5544 cs=0 l=1 rev1=1 crypto rx=0x7f912c00e9b0 tx=0x7f912c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.088+0000 7f91267fc640 10 client.?.objecter ms_handle_connect 0x7f91300a9c10 2024-03-25T02:35:13.080 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.088+0000 7f91267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c017070 con 0x7f91300a9c10 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: dump: 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.082 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: _finish_auth 0 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: _check_auth_tickets 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c00c9a0 con 0x7f91300a9c10 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 5 monclient: authenticate success, global_id 22081 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 10 monclient: _renew_subs 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91301341a0 con 0x7f91300a9c10 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c00cd90 con 0x7f91300a9c10 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.083 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f91267fc640 10 monclient: dump: 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9127fff640 4 set_mon_vals no callback set 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 10 client.22081.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 10 monclient: _renew_subs 2024-03-25T02:35:13.084 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9130134de0 con 0x7f91300a9c10 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.091+0000 7f9137e84640 1 librados: init done 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.092+0000 7f9137e84640 10 client.22081.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.092+0000 7f9137e84640 15 client.22081.objecter _session_command_op_assign -1 1 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.092+0000 7f9137e84640 20 client.22081.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.092+0000 7f9137e84640 10 client.22081.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.092+0000 7f9137e84640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:13.085 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.092+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9130076d20 con 0x7f91300a9c10 2024-03-25T02:35:13.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.110+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f912c01a070 con 0x7f91300a9c10 2024-03-25T02:35:13.099 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.110+0000 7f91267fc640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f911406da70 0x7f911406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.111+0000 7f913541f640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f911406da70 0x7f911406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.111+0000 7f913541f640 10 monclient: get_auth_request con 0x7f911406da70 auth_method 0 2024-03-25T02:35:13.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.111+0000 7f913541f640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f911406da70 0x7f911406ff30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9130142620 tx=0x7f9120009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.100 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.111+0000 7f91267fc640 10 client.22081.objecter ms_handle_connect 0x7f911406da70 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f912c057690 con 0x7f91300a9c10 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 10 client.22081.objecter ms_dispatch 0x7f9130000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 3 client.22081.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 3 client.22081.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 10 client.22081.objecter checking command 1 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f91140757d0 0x7f9114077c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 20 client.22081.objecter _get_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 20 client.22081.objecter put_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.102 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 15 client.22081.objecter _session_command_op_remove -1 1 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 20 client.22081.objecter _get_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 20 client.22081.objecter get_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 15 client.22081.objecter _session_command_op_assign 5 1 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 20 client.22081.objecter put_session s=0x7f9114075100 osd=5 4 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 10 client.22081.objecter _send_command 1 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 --> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f912c020e50 con 0x7f91140757d0 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.113+0000 7f91267fc640 20 client.22081.objecter dump_active .. 0 homeless 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.114+0000 7f9136421640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f91140757d0 0x7f9114077c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.103 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.114+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=1250) ==== 24+0+0 (secure 0 0 0) 0x7f912c055890 con 0x7f91300a9c10 2024-03-25T02:35:13.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.114+0000 7f91267fc640 10 monclient: handle_get_version_reply finishing 1 version 1250 2024-03-25T02:35:13.104 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.115+0000 7f9136421640 10 monclient: get_auth_request con 0x7f91140757d0 auth_method 0 2024-03-25T02:35:13.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.117+0000 7f9136421640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f91140757d0 0x7f9114077c70 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.106 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.117+0000 7f91267fc640 10 client.22081.objecter ms_handle_connect 0x7f91140757d0 2024-03-25T02:35:13.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.140+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== osd.5 v2:172.21.15.153:6810/2458137144 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f912c020e50 con 0x7f91140757d0 2024-03-25T02:35:13.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.140+0000 7f91267fc640 10 client.22081.objecter ms_dispatch 0x7f9130000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:13.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.140+0000 7f91267fc640 10 client.22081.objecter _finish_command 1 = system:0 2024-03-25T02:35:13.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.140+0000 7f91267fc640 20 client.22081.objecter put_session s=0x7f9114075100 osd=5 4 2024-03-25T02:35:13.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.140+0000 7f91267fc640 15 client.22081.objecter _session_command_op_remove 5 1 2024-03-25T02:35:13.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 10 client.22081.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:13.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 15 client.22081.objecter _session_command_op_assign -1 2 2024-03-25T02:35:13.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 20 client.22081.objecter _get_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 20 client.22081.objecter put_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 20 client.22081.objecter _get_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 15 client.22081.objecter _session_command_op_remove -1 2 2024-03-25T02:35:13.142 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 20 client.22081.objecter get_session s=0x7f9114075100 osd=5 3 2024-03-25T02:35:13.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 15 client.22081.objecter _session_command_op_assign 5 2 2024-03-25T02:35:13.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 20 client.22081.objecter put_session s=0x7f9114075100 osd=5 4 2024-03-25T02:35:13.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 10 client.22081.objecter _send_command 2 2024-03-25T02:35:13.143 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.153+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 --> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9130075810 con 0x7f91140757d0 2024-03-25T02:35:13.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f91267fc640 1 -- 172.21.15.83:0/1862664057 <== osd.5 v2:172.21.15.153:6810/2458137144 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9130075810 con 0x7f91140757d0 2024-03-25T02:35:13.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f91267fc640 10 client.22081.objecter ms_dispatch 0x7f9130000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:13.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f91267fc640 10 client.22081.objecter _finish_command 2 = system:0 2024-03-25T02:35:13.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f91267fc640 20 client.22081.objecter put_session s=0x7f9114075100 osd=5 4 2024-03-25T02:35:13.155 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f91267fc640 15 client.22081.objecter _session_command_op_remove 5 2 2024-03-25T02:35:13.155 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:13.156 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:13.156 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 10 librados: watch_flush enter 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 10 librados: watch_flush exit 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 10 client.22081.objecter close_session for osd.5 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f91140757d0 msgr2=0x7f9114077c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6810/2458137144,v1:172.21.15.153:6811/2458137144] conn(0x7f91140757d0 0x7f9114077c70 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 20 client.22081.objecter put_session s=0x7f9114075100 osd=5 1 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 20 client.22081.objecter shutdown clearing up homeless session... 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 10 client.22081.objecter successfully canceled tick 2024-03-25T02:35:13.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f911406da70 msgr2=0x7f911406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f911406da70 0x7f911406ff30 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9130142620 tx=0x7f9120009040 comp rx=0 tx=0).stop 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 10 monclient: shutdown 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91300a9c10 msgr2=0x7f91301420e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 --2- 172.21.15.83:0/1862664057 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f91300a9c10 0x7f91301420e0 secure :-1 s=READY pgs=5544 cs=0 l=1 rev1=1 crypto rx=0x7f912c00e9b0 tx=0x7f912c00ee70 comp rx=0 tx=0).stop 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9136421640 1 -- 172.21.15.83:0/1862664057 reap_dead start 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 shutdown_connections 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 >> 172.21.15.83:0/1862664057 conn(0x7f91300b8a60 msgr2=0x7f91300b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 shutdown_connections 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 -- 172.21.15.83:0/1862664057 wait complete. 2024-03-25T02:35:13.158 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.167+0000 7f9137e84640 1 librados: shutdown 2024-03-25T02:35:13.166 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-25T02:35:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.281+0000 7f197356d640 10 monclient: get_monmap_and_config 2024-03-25T02:35:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.281+0000 7f197356d640 10 monclient: build_initial_monmap 2024-03-25T02:35:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.281+0000 7f197356d640 10 monclient: monmap: 2024-03-25T02:35:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.270 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:13.282513+0000 2024-03-25T02:35:13.271 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:13.282513+0000 2024-03-25T02:35:13.271 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:13.272 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:13.272 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.281+0000 7f197356d640 1 Processor -- start 2024-03-25T02:35:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.281+0000 7f197356d640 1 -- start start 2024-03-25T02:35:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.281+0000 7f197356d640 10 monclient: init 2024-03-25T02:35:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-03-25T02:35:13.274 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f196c0af4a0 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:13.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f196c0706f0 0x7f196c143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient(hunting): picked mon.noname-c con 0x7f196c0706f0 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:13.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f196c1438d0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:13.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f196c0710a0 con 0x7f196c1438d0 2024-03-25T02:35:13.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f196c072e90 con 0x7f196c0af4a0 2024-03-25T02:35:13.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.275 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f196c0af880 con 0x7f196c0706f0 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.282+0000 7f197356d640 10 monclient(hunting): authenticate will time out at 3432.472528s 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.283+0000 7f1971b0a640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f196c1438d0 0x7f196c14a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:54242/0 (socket says 172.21.15.83:54242) 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.283+0000 7f1971b0a640 1 -- 172.21.15.83:0/4086786461 learned_addr learned my addr 172.21.15.83:0/4086786461 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.283+0000 7f1970b08640 1 --2- 172.21.15.83:0/4086786461 >> v2:172.21.15.153:3300/0 conn(0x7f196c0706f0 0x7f196c143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.283+0000 7f1970b08640 10 monclient(hunting): get_auth_request con 0x7f196c0706f0 auth_method 0 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.283+0000 7f1970b08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.283+0000 7f1970b08640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.283+0000 7f1970b08640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.284+0000 7f1970b08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.284+0000 7f1970b08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.284+0000 7f1970b08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.285+0000 7f1963fff640 1 -- 172.21.15.83:0/4086786461 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780550695 0 0) 0x7f196c0710a0 con 0x7f196c1438d0 2024-03-25T02:35:13.276 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.285+0000 7f1963fff640 10 monclient(hunting): _negotiate 2024-03-25T02:35:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.285+0000 7f1963fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.285+0000 7f1963fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.285+0000 7f1963fff640 10 monclient(hunting): my global_id is 15426 2024-03-25T02:35:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.285+0000 7f1963fff640 1 -- 172.21.15.83:0/4086786461 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1958003520 con 0x7f196c1438d0 2024-03-25T02:35:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 10 monclient(hunting): handle_auth_done global_id 22084 payload 274 2024-03-25T02:35:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 1 -- 172.21.15.83:0/4086786461 >> v1:172.21.15.153:6789/0 conn(0x7f196c0af4a0 legacy=0x7f196c13d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.277 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 1 -- 172.21.15.83:0/4086786461 >> v1:172.21.15.83:6789/0 conn(0x7f196c1438d0 legacy=0x7f196c14a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 10 monclient: _finish_hunting 0 2024-03-25T02:35:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 1 monclient: found mon.noname-c 2024-03-25T02:35:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:35:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.286+0000 7f1970b08640 1 -- 172.21.15.83:0/4086786461 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196c071790 con 0x7f196c0706f0 2024-03-25T02:35:13.278 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.290+0000 7f1970b08640 1 --2- 172.21.15.83:0/4086786461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c0706f0 0x7f196c143390 secure :-1 s=READY pgs=5546 cs=0 l=1 rev1=1 crypto rx=0x7f195c0097a0 tx=0x7f195c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f825be230b59413 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.294+0000 7f1963fff640 1 -- 172.21.15.83:0/4086786461 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c034070 con 0x7f196c0706f0 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.294+0000 7f1963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.294+0000 7f1963fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.294+0000 7f1963fff640 10 monclient: dump: 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.284 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 10 monclient: _finish_auth 0 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 10 monclient: _check_auth_tickets 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 1 -- 172.21.15.83:0/4086786461 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f195c030610 con 0x7f196c0706f0 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 5 monclient: authenticate success, global_id 22084 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 4 set_mon_vals no callback set 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 1 -- 172.21.15.83:0/4086786461 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c0309e0 con 0x7f196c0706f0 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f1963fff640 10 monclient: dump: 2024-03-25T02:35:13.285 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.286 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 10 monclient: shutdown 2024-03-25T02:35:13.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:13.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 1 -- 172.21.15.83:0/4086786461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c0706f0 msgr2=0x7f196c143390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 1 --2- 172.21.15.83:0/4086786461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c0706f0 0x7f196c143390 secure :-1 s=READY pgs=5546 cs=0 l=1 rev1=1 crypto rx=0x7f195c0097a0 tx=0x7f195c02b960 comp rx=0 tx=0).stop 2024-03-25T02:35:13.289 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 1 -- 172.21.15.83:0/4086786461 shutdown_connections 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 1 --2- 172.21.15.83:0/4086786461 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c0706f0 0x7f196c143390 unknown :-1 s=CLOSED pgs=5546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 1 -- 172.21.15.83:0/4086786461 >> 172.21.15.83:0/4086786461 conn(0x7f196c0b4a10 msgr2=0x7f196c0b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 1 -- 172.21.15.83:0/4086786461 shutdown_connections 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.295+0000 7f197356d640 1 -- 172.21.15.83:0/4086786461 wait complete. 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 10 monclient: build_initial_monmap 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 10 monclient: monmap: 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:13.290 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 1 librados: starting msgr at 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 1 librados: starting objecter 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 1 Processor -- start 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 1 -- start start 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 1 librados: setting wanted keys 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 1 librados: calling monclient init 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.296+0000 7f197356d640 10 monclient: init 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f196c0af4a0 0x7f196c139880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): picked mon.noname-a con 0x7f196c0af4a0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c1438d0 0x7f196c0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.291 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): picked mon.noname-b con 0x7f196c1438d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f196c13fa50 0x7f196c140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): picked mon.noname-c con 0x7f196c13fa50 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f196c143d20 con 0x7f196c0af4a0 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f196c070b80 con 0x7f196c1438d0 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f196c13d8d0 con 0x7f196c13fa50 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f197356d640 10 monclient(hunting): authenticate will time out at 3432.487476s 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.297+0000 7f1971309640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f196c0af4a0 0x7f196c139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971b0a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f196c13fa50 0x7f196c140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c1438d0 0x7f196c0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f196c0af4a0 0x7f196c139880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:49664/0 (socket says 172.21.15.83:49664) 2024-03-25T02:35:13.292 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 1 -- 172.21.15.83:0/2780016488 learned_addr learned my addr 172.21.15.83:0/2780016488 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 10 monclient(hunting): get_auth_request con 0x7f196c0af4a0 auth_method 0 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971b0a640 10 monclient(hunting): get_auth_request con 0x7f196c13fa50 auth_method 0 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971b0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971b0a640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971b0a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 10 monclient(hunting): get_auth_request con 0x7f196c1438d0 auth_method 0 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.293 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1971309640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.298+0000 7f1970b08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.299+0000 7f1971b0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.299+0000 7f1971b0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.294 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.299+0000 7f1971b0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 10 monclient(hunting): handle_auth_done global_id 15429 payload 824 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 1 -- 172.21.15.83:0/2780016488 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f196c13fa50 msgr2=0x7f196c140580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f196c13fa50 0x7f196c140580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 1 -- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c1438d0 msgr2=0x7f196c0ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c1438d0 0x7f196c0ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 10 monclient: _finish_hunting 0 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 1 monclient: found mon.noname-a 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.301+0000 7f1971309640 1 -- 172.21.15.83:0/2780016488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196c142120 con 0x7f196c0af4a0 2024-03-25T02:35:13.295 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.303+0000 7f1971309640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f196c0af4a0 0x7f196c139880 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f196800c480 tx=0x7f196800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.303+0000 7f1961ffb640 10 client.?.objecter ms_handle_connect 0x7f196c0af4a0 2024-03-25T02:35:13.296 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.303+0000 7f1961ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 1 -- 172.21.15.83:0/2780016488 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680052d0 con 0x7f196c0af4a0 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: dump: 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.297 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: _finish_auth 0 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: _check_auth_tickets 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 1 -- 172.21.15.83:0/2780016488 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1968005470 con 0x7f196c0af4a0 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 5 monclient: authenticate success, global_id 15429 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 10 monclient: _renew_subs 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 1 -- 172.21.15.83:0/2780016488 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968012870 con 0x7f196c0af4a0 2024-03-25T02:35:13.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196c142590 con 0x7f196c0af4a0 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f19637fe640 4 set_mon_vals no callback set 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f1961ffb640 10 monclient: dump: 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.299 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 10 client.15429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 10 monclient: _renew_subs 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f196c140db0 con 0x7f196c0af4a0 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.307+0000 7f197356d640 1 librados: init done 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.308+0000 7f197356d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.308+0000 7f197356d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.308+0000 7f197356d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.308+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f196c075810 con 0x7f196c0af4a0 2024-03-25T02:35:13.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.557+0000 7f1961ffb640 1 -- 172.21.15.83:0/2780016488 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f1968012a10 con 0x7f196c0af4a0 2024-03-25T02:35:13.546 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.557+0000 7f1961ffb640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f195006da20 0x7f195006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.558+0000 7f1970b08640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f195006da20 0x7f195006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.558+0000 7f1970b08640 10 monclient: get_auth_request con 0x7f195006da20 auth_method 0 2024-03-25T02:35:13.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.558+0000 7f1970b08640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f195006da20 0x7f195006fee0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f195c036000 tx=0x7f195c03b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.547 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.558+0000 7f1961ffb640 10 client.15429.objecter ms_handle_connect 0x7f195006da20 2024-03-25T02:35:13.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 1 -- 172.21.15.83:0/2780016488 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f19680579d0 con 0x7f196c0af4a0 2024-03-25T02:35:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 10 client.15429.objecter ms_dispatch 0x7f196c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 3 client.15429.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 3 client.15429.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 20 client.15429.objecter dump_active .. 0 homeless 2024-03-25T02:35:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 1 -- 172.21.15.83:0/2780016488 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1968059930 con 0x7f196c0af4a0 2024-03-25T02:35:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:13.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.570+0000 7f1961ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:13.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.758+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f196c076d20 con 0x7f195006da20 2024-03-25T02:35:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.759+0000 7f1961ffb640 1 -- 172.21.15.83:0/2780016488 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f195c039620 con 0x7f195006da20 2024-03-25T02:35:13.748 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 5 to deep-scrub 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 10 librados: watch_flush enter 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 10 librados: watch_flush exit 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 20 client.15429.objecter shutdown clearing up homeless session... 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 10 client.15429.objecter successfully canceled tick 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f195006da20 msgr2=0x7f195006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f195006da20 0x7f195006fee0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f195c036000 tx=0x7f195c03b040 comp rx=0 tx=0).stop 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 10 monclient: shutdown 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f196c0af4a0 msgr2=0x7f196c139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f196c0af4a0 0x7f196c139880 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f196800c480 tx=0x7f196800c940 comp rx=0 tx=0).stop 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 shutdown_connections 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f195006da20 0x7f195006fee0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f196c13fa50 0x7f196c140580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f196c1438d0 0x7f196c0ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 --2- 172.21.15.83:0/2780016488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f196c0af4a0 0x7f196c139880 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 >> 172.21.15.83:0/2780016488 conn(0x7f196c0b4a10 msgr2=0x7f196c13ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 shutdown_connections 2024-03-25T02:35:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 -- 172.21.15.83:0/2780016488 wait complete. 2024-03-25T02:35:13.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.761+0000 7f197356d640 1 librados: shutdown 2024-03-25T02:35:13.761 INFO:tasks.ceph:Scrubbing osd.6 2024-03-25T02:35:13.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.884+0000 7fe780de2640 10 monclient: get_monmap_and_config 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.884+0000 7fe780de2640 10 monclient: build_initial_monmap 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.884+0000 7fe780de2640 10 monclient: monmap: 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:13.886336+0000 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:13.886336+0000 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:13.874 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:13.875 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:13.875 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:13.875 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:13.875 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.885+0000 7fe780de2640 1 Processor -- start 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.885+0000 7fe780de2640 1 -- start start 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.885+0000 7fe780de2640 10 monclient: init 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7fe77c0a9c10 0x7fe77c0a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient(hunting): picked mon.noname-a con 0x7fe77c0a9c10 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe77c1403b0 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:13.876 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe77c143c90 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe77c0abe60 con 0x7fe77c143c90 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe77c0ac140 con 0x7fe77c1403b0 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7fe77c1430f0 con 0x7fe77c0a9c10 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.886+0000 7fe780de2640 10 monclient(hunting): authenticate will time out at 3433.076631s 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.888+0000 7fe773fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe77c1403b0 0x7fe77c143580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:38148/0 (socket says 172.21.15.83:38148) 2024-03-25T02:35:13.877 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.888+0000 7fe773fff640 1 -- 172.21.15.83:0/4127220011 learned_addr learned my addr 172.21.15.83:0/4127220011 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:13.879 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.890+0000 7fe77a59c640 1 --2- 172.21.15.83:0/4127220011 >> v2:172.21.15.83:3300/0 conn(0x7fe77c0a9c10 0x7fe77c0a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.891+0000 7fe77a59c640 10 monclient(hunting): get_auth_request con 0x7fe77c0a9c10 auth_method 0 2024-03-25T02:35:13.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.891+0000 7fe77a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.891+0000 7fe77a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.880 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.891+0000 7fe77a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.892+0000 7fe779d9b640 1 -- 172.21.15.83:0/4127220011 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658819282 0 0) 0x7fe77c0ac140 con 0x7fe77c1403b0 2024-03-25T02:35:13.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.892+0000 7fe779d9b640 10 monclient(hunting): _negotiate 2024-03-25T02:35:13.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.892+0000 7fe779d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.892+0000 7fe779d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.892+0000 7fe779d9b640 10 monclient(hunting): my global_id is 22093 2024-03-25T02:35:13.881 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.892+0000 7fe779d9b640 1 -- 172.21.15.83:0/4127220011 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe768003520 con 0x7fe77c1403b0 2024-03-25T02:35:13.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.893+0000 7fe77a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.893+0000 7fe77a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.882 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.893+0000 7fe77a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 10 monclient(hunting): handle_auth_done global_id 15432 payload 274 2024-03-25T02:35:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 1 -- 172.21.15.83:0/4127220011 >> v1:172.21.15.153:6789/0 conn(0x7fe77c1403b0 legacy=0x7fe77c143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 1 -- 172.21.15.83:0/4127220011 >> v1:172.21.15.83:6789/0 conn(0x7fe77c143c90 legacy=0x7fe77c14d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 10 monclient: _finish_hunting 0 2024-03-25T02:35:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 1 monclient: found mon.noname-a 2024-03-25T02:35:13.887 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:13.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.888 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.898+0000 7fe77a59c640 1 -- 172.21.15.83:0/4127220011 --> v2:172.21.15.83:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77c0b3460 con 0x7fe77c0a9c10 2024-03-25T02:35:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.903+0000 7fe77a59c640 1 --2- 172.21.15.83:0/4127220011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c0a9c10 0x7fe77c0a9ff0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0097a0 tx=0x7fe76c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c525e88367708 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 1 -- 172.21.15.83:0/4127220011 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c035020 con 0x7fe77c0a9c10 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: dump: 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.944 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: _finish_auth 0 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: _check_auth_tickets 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 5 monclient: authenticate success, global_id 15432 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 1 -- 172.21.15.83:0/4127220011 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe76c031830 con 0x7fe77c0a9c10 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 4 set_mon_vals no callback set 2024-03-25T02:35:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 1 -- 172.21.15.83:0/4127220011 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c031c00 con 0x7fe77c0a9c10 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe779d9b640 10 monclient: dump: 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.946 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 10 monclient: shutdown 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 1 -- 172.21.15.83:0/4127220011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c0a9c10 msgr2=0x7fe77c0a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 1 --2- 172.21.15.83:0/4127220011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c0a9c10 0x7fe77c0a9ff0 secure :-1 s=READY pgs=3213 cs=0 l=1 rev1=1 crypto rx=0x7fe76c0097a0 tx=0x7fe76c02b8d0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 1 -- 172.21.15.83:0/4127220011 shutdown_connections 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 1 --2- 172.21.15.83:0/4127220011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c0a9c10 0x7fe77c0a9ff0 unknown :-1 s=CLOSED pgs=3213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 1 -- 172.21.15.83:0/4127220011 >> 172.21.15.83:0/4127220011 conn(0x7fe77c0b8a60 msgr2=0x7fe77c0b8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 1 -- 172.21.15.83:0/4127220011 shutdown_connections 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.955+0000 7fe780de2640 1 -- 172.21.15.83:0/4127220011 wait complete. 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.956+0000 7fe780de2640 10 monclient: build_initial_monmap 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.956+0000 7fe780de2640 10 monclient: monmap: 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.950 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.956+0000 7fe780de2640 1 librados: starting msgr at 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.956+0000 7fe780de2640 1 librados: starting objecter 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 Processor -- start 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 -- start start 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 librados: setting wanted keys 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 librados: calling monclient init 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient: init 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe77c0a9c10 0x7fe77c142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): picked mon.noname-b con 0x7fe77c0a9c10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c1403b0 0x7fe77c0b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): picked mon.noname-a con 0x7fe77c1403b0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe77c143c90 0x7fe77c0b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.951 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): picked mon.noname-c con 0x7fe77c143c90 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c140990 con 0x7fe77c1403b0 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c140810 con 0x7fe77c0a9c10 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fe77c140b10 con 0x7fe77c143c90 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.957+0000 7fe780de2640 10 monclient(hunting): authenticate will time out at 3433.147689s 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.958+0000 7fe77ad9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe77c143c90 0x7fe77c0b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.959+0000 7fe77ad9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe77c143c90 0x7fe77c0b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:43286/0 (socket says 172.21.15.83:43286) 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.959+0000 7fe77ad9d640 1 -- 172.21.15.83:0/1915998507 learned_addr learned my addr 172.21.15.83:0/1915998507 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.959+0000 7fe77ad9d640 10 monclient(hunting): get_auth_request con 0x7fe77c143c90 auth_method 0 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.959+0000 7fe77ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.952 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.959+0000 7fe77ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.959+0000 7fe77ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.959+0000 7fe773fff640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c1403b0 0x7fe77c0b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.960+0000 7fe773fff640 10 monclient(hunting): get_auth_request con 0x7fe77c1403b0 auth_method 0 2024-03-25T02:35:13.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.960+0000 7fe773fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:13.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.960+0000 7fe773fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:13.953 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.960+0000 7fe773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.960+0000 7fe773fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.960+0000 7fe773fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.960+0000 7fe773fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.962+0000 7fe77ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.962+0000 7fe77ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.962+0000 7fe77ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 10 monclient(hunting): handle_auth_done global_id 15438 payload 824 2024-03-25T02:35:13.954 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 1 -- 172.21.15.83:0/1915998507 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe77c143c90 msgr2=0x7fe77c0b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe77c143c90 0x7fe77c0b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 1 -- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe77c0a9c10 msgr2=0x7fe77c142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe77c0a9c10 0x7fe77c142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 1 monclient: found mon.noname-a 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.963+0000 7fe773fff640 1 -- 172.21.15.83:0/1915998507 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77c0b2910 con 0x7fe77c1403b0 2024-03-25T02:35:13.955 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.966+0000 7fe773fff640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c1403b0 0x7fe77c0b1e90 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c430 tx=0x7fe76400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.966+0000 7fe772ffd640 10 client.?.objecter ms_handle_connect 0x7fe77c1403b0 2024-03-25T02:35:13.956 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.966+0000 7fe772ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe764017070 con 0x7fe77c1403b0 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: dump: 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.959 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: _finish_auth 0 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe764005ce0 con 0x7fe77c1403b0 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7640072d0 con 0x7fe77c1403b0 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 5 monclient: authenticate success, global_id 15438 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 10 monclient: _renew_subs 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77c146190 con 0x7fe77c1403b0 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:13.960 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe778d99640 4 set_mon_vals no callback set 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe772ffd640 10 monclient: dump: 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 10 client.15438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 10 monclient: _renew_subs 2024-03-25T02:35:13.961 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe77c1471b0 con 0x7fe77c1403b0 2024-03-25T02:35:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.969+0000 7fe780de2640 1 librados: init done 2024-03-25T02:35:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.970+0000 7fe780de2640 10 client.15438.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.970+0000 7fe780de2640 15 client.15438.objecter _session_command_op_assign -1 1 2024-03-25T02:35:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.970+0000 7fe780de2640 20 client.15438.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.970+0000 7fe780de2640 10 client.15438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.970+0000 7fe780de2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:13.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.970+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe77c076d20 con 0x7fe77c1403b0 2024-03-25T02:35:13.985 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.995+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fe76401a070 con 0x7fe77c1403b0 2024-03-25T02:35:13.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.995+0000 7fe772ffd640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe75c06da70 0x7fe75c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.996+0000 7fe77a59c640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe75c06da70 0x7fe75c06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.996+0000 7fe77a59c640 10 monclient: get_auth_request con 0x7fe75c06da70 auth_method 0 2024-03-25T02:35:13.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.996+0000 7fe772ffd640 10 client.15438.objecter ms_handle_connect 0x7fe75c06da70 2024-03-25T02:35:13.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.996+0000 7fe77a59c640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe75c06da70 0x7fe75c06ff30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe76c036000 tx=0x7fe76c030000 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7fe764056e90 con 0x7fe77c1403b0 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 10 client.15438.objecter ms_dispatch 0x7fe77c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 3 client.15438.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 3 client.15438.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 10 client.15438.objecter checking command 1 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7fe75c075770 0x7fe75c077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 20 client.15438.objecter _get_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:13.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 20 client.15438.objecter put_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 15 client.15438.objecter _session_command_op_remove -1 1 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 20 client.15438.objecter _get_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 20 client.15438.objecter get_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 15 client.15438.objecter _session_command_op_assign 6 1 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 20 client.15438.objecter put_session s=0x7fe75c0750a0 osd=6 4 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 10 client.15438.objecter _send_command 1 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 --> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe75c0782e0 con 0x7fe75c075770 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 20 client.15438.objecter dump_active .. 0 homeless 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=1250) ==== 24+0+0 (secure 0 0 0) 0x7fe764055090 con 0x7fe77c1403b0 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:13.999+0000 7fe772ffd640 10 monclient: handle_get_version_reply finishing 1 version 1250 2024-03-25T02:35:13.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.000+0000 7fe77ad9d640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7fe75c075770 0x7fe75c077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:13.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.000+0000 7fe77ad9d640 10 monclient: get_auth_request con 0x7fe75c075770 auth_method 0 2024-03-25T02:35:13.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.003+0000 7fe772ffd640 10 client.15438.objecter ms_handle_connect 0x7fe75c075770 2024-03-25T02:35:13.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.003+0000 7fe77ad9d640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7fe75c075770 0x7fe75c077c10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.027+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== osd.6 v2:172.21.15.153:6802/4205760170 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe75c0782e0 con 0x7fe75c075770 2024-03-25T02:35:14.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.027+0000 7fe772ffd640 10 client.15438.objecter ms_dispatch 0x7fe77c000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:14.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.027+0000 7fe772ffd640 10 client.15438.objecter _finish_command 1 = system:0 2024-03-25T02:35:14.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.027+0000 7fe772ffd640 20 client.15438.objecter put_session s=0x7fe75c0750a0 osd=6 4 2024-03-25T02:35:14.017 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.027+0000 7fe772ffd640 15 client.15438.objecter _session_command_op_remove 6 1 2024-03-25T02:35:14.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 10 client.15438.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:14.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 15 client.15438.objecter _session_command_op_assign -1 2 2024-03-25T02:35:14.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 20 client.15438.objecter _get_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:14.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 20 client.15438.objecter put_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:14.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 20 client.15438.objecter _get_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:14.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 15 client.15438.objecter _session_command_op_remove -1 2 2024-03-25T02:35:14.030 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 20 client.15438.objecter get_session s=0x7fe75c0750a0 osd=6 3 2024-03-25T02:35:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 15 client.15438.objecter _session_command_op_assign 6 2 2024-03-25T02:35:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 20 client.15438.objecter put_session s=0x7fe75c0750a0 osd=6 4 2024-03-25T02:35:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 10 client.15438.objecter _send_command 2 2024-03-25T02:35:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.040+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 --> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe77c075810 con 0x7fe75c075770 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe772ffd640 1 -- 172.21.15.83:0/1915998507 <== osd.6 v2:172.21.15.153:6802/4205760170 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe77c075810 con 0x7fe75c075770 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe772ffd640 10 client.15438.objecter ms_dispatch 0x7fe77c000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe772ffd640 10 client.15438.objecter _finish_command 2 = system:0 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe772ffd640 20 client.15438.objecter put_session s=0x7fe75c0750a0 osd=6 4 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe772ffd640 15 client.15438.objecter _session_command_op_remove 6 2 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:14.043 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 10 librados: watch_flush enter 2024-03-25T02:35:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 10 librados: watch_flush exit 2024-03-25T02:35:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 10 client.15438.objecter close_session for osd.6 2024-03-25T02:35:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7fe75c075770 msgr2=0x7fe75c077c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6802/4205760170,v1:172.21.15.153:6803/4205760170] conn(0x7fe75c075770 0x7fe75c077c10 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 20 client.15438.objecter put_session s=0x7fe75c0750a0 osd=6 1 2024-03-25T02:35:14.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 20 client.15438.objecter shutdown clearing up homeless session... 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 10 client.15438.objecter successfully canceled tick 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe75c06da70 msgr2=0x7fe75c06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe75c06da70 0x7fe75c06ff30 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe76c036000 tx=0x7fe76c030000 comp rx=0 tx=0).stop 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 10 monclient: shutdown 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c1403b0 msgr2=0x7fe77c0b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 --2- 172.21.15.83:0/1915998507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe77c1403b0 0x7fe77c0b1e90 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7fe76400c430 tx=0x7fe76400c8f0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe77ad9d640 1 -- 172.21.15.83:0/1915998507 reap_dead start 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 shutdown_connections 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 >> 172.21.15.83:0/1915998507 conn(0x7fe77c0b8a60 msgr2=0x7fe77c0b8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 shutdown_connections 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 -- 172.21.15.83:0/1915998507 wait complete. 2024-03-25T02:35:14.046 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.054+0000 7fe780de2640 1 librados: shutdown 2024-03-25T02:35:14.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.168+0000 7f52202c5640 10 monclient: get_monmap_and_config 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.168+0000 7f52202c5640 10 monclient: build_initial_monmap 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.168+0000 7f52202c5640 10 monclient: monmap: 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:14.169624+0000 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:14.169624+0000 2024-03-25T02:35:14.157 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:14.158 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:14.158 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.168+0000 7f52202c5640 1 Processor -- start 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.168+0000 7f52202c5640 1 -- start start 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.168+0000 7f52202c5640 10 monclient: init 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52180b04a0 0x7f5218146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient(hunting): picked mon.noname-a con 0x7f52180b04a0 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:14.160 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52180b2df0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f5218140b10 0x7f5218147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient(hunting): picked mon.noname-b con 0x7f5218140b10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52180b2410 con 0x7f52180b2df0 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f52180b26f0 con 0x7f52180b04a0 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f52180b29d0 con 0x7f5218140b10 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.169+0000 7f52202c5640 10 monclient(hunting): authenticate will time out at 3433.359500s 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e862640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f5218140b10 0x7f5218147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e061640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52180b04a0 0x7f5218146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e061640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f52180b04a0 0x7f5218146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33016/0 (socket says 172.21.15.83:33016) 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e061640 1 -- 172.21.15.83:0/3675601831 learned_addr learned my addr 172.21.15.83:0/3675601831 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e061640 10 monclient(hunting): get_auth_request con 0x7f52180b04a0 auth_method 0 2024-03-25T02:35:14.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e862640 1 --2- 172.21.15.83:0/3675601831 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f5218140b10 0x7f5218147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:43290/0 (socket says 172.21.15.83:43290) 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e061640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e061640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e061640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e862640 10 monclient(hunting): get_auth_request con 0x7f5218140b10 auth_method 0 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e862640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.170+0000 7f521e862640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.171+0000 7f521e862640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.171+0000 7f521e862640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.171+0000 7f521e862640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.171+0000 7f521e061640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.171+0000 7f521e061640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.162 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.171+0000 7f521e061640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 10 monclient(hunting): handle_auth_done global_id 17276 payload 274 2024-03-25T02:35:14.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 1 -- 172.21.15.83:0/3675601831 >> v2:172.21.15.83:3300/0 conn(0x7f52180b04a0 msgr2=0x7f5218146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:14.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 1 --2- 172.21.15.83:0/3675601831 >> v2:172.21.15.83:3300/0 conn(0x7f52180b04a0 0x7f5218146c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 1 -- 172.21.15.83:0/3675601831 >> v1:172.21.15.83:6789/0 conn(0x7f52180b2df0 legacy=0x7f5218140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:14.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 10 monclient: _finish_hunting 0 2024-03-25T02:35:14.163 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 1 monclient: found mon.noname-b 2024-03-25T02:35:14.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:14.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:14.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.172+0000 7f521e862640 1 -- 172.21.15.83:0/3675601831 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52180b3380 con 0x7f5218140b10 2024-03-25T02:35:14.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.174+0000 7f521e862640 1 --2- 172.21.15.83:0/3675601831 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f5218140b10 0x7f5218147180 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f52140097d0 tx=0x7f521402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=691462194e18da41 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f521d05f640 1 -- 172.21.15.83:0/3675601831 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214035020 con 0x7f5218140b10 2024-03-25T02:35:14.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f521d05f640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f521d05f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:14.165 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f521d05f640 10 monclient: dump: 2024-03-25T02:35:14.165 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f521d05f640 10 monclient: _finish_auth 0 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f521d05f640 10 monclient: _check_auth_tickets 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f521d05f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f52202c5640 5 monclient: authenticate success, global_id 17276 2024-03-25T02:35:14.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.176+0000 7f52202c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f521d05f640 1 -- 172.21.15.83:0/3675601831 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5214031800 con 0x7f5218140b10 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f521d05f640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f521d05f640 1 -- 172.21.15.83:0/3675601831 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214031bd0 con 0x7f5218140b10 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f521d05f640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f521d05f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f521d05f640 10 monclient: dump: 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.168 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f52202c5640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f52202c5640 4 set_mon_vals no callback set 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f52202c5640 10 monclient: shutdown 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f52202c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f52202c5640 1 -- 172.21.15.83:0/3675601831 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f5218140b10 msgr2=0x7f5218147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.178+0000 7f52202c5640 1 --2- 172.21.15.83:0/3675601831 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f5218140b10 0x7f5218147180 secure :-1 s=READY pgs=3205 cs=0 l=1 rev1=1 crypto rx=0x7f52140097d0 tx=0x7f521402b8a0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 1 -- 172.21.15.83:0/3675601831 shutdown_connections 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 1 --2- 172.21.15.83:0/3675601831 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f5218140b10 0x7f5218147180 unknown :-1 s=CLOSED pgs=3205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.169 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 1 --2- 172.21.15.83:0/3675601831 >> v2:172.21.15.83:3300/0 conn(0x7f52180b04a0 0x7f5218146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 1 -- 172.21.15.83:0/3675601831 >> 172.21.15.83:0/3675601831 conn(0x7f52180b8a00 msgr2=0x7f52180b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:14.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 1 -- 172.21.15.83:0/3675601831 shutdown_connections 2024-03-25T02:35:14.170 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 1 -- 172.21.15.83:0/3675601831 wait complete. 2024-03-25T02:35:14.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 10 monclient: build_initial_monmap 2024-03-25T02:35:14.172 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.179+0000 7f52202c5640 10 monclient: monmap: 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 librados: starting msgr at 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 librados: starting objecter 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 Processor -- start 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 -- start start 2024-03-25T02:35:14.173 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 librados: setting wanted keys 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 librados: calling monclient init 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient: init 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52180b04a0 0x7f5218138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): picked mon.noname-b con 0x7f52180b04a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f52180b2df0 0x7f5218144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): picked mon.noname-c con 0x7f52180b2df0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5218140b10 0x7f5218144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): picked mon.noname-a con 0x7f5218140b10 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f521814dc70 con 0x7f5218140b10 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f521814daf0 con 0x7f52180b04a0 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f521814ddf0 con 0x7f52180b2df0 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.180+0000 7f52202c5640 10 monclient(hunting): authenticate will time out at 3433.370887s 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521d860640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f52180b2df0 0x7f5218144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e862640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5218140b10 0x7f5218144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e061640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52180b04a0 0x7f5218138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521d860640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f52180b2df0 0x7f5218144560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:43298/0 (socket says 172.21.15.83:43298) 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521d860640 1 -- 172.21.15.83:0/1564584640 learned_addr learned my addr 172.21.15.83:0/1564584640 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521d860640 10 monclient(hunting): get_auth_request con 0x7f52180b2df0 auth_method 0 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521d860640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521d860640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521d860640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e862640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5218140b10 0x7f5218144aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33026/0 (socket says 172.21.15.83:33026) 2024-03-25T02:35:14.175 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e862640 10 monclient(hunting): get_auth_request con 0x7f5218140b10 auth_method 0 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e862640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e862640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e061640 10 monclient(hunting): get_auth_request con 0x7f52180b04a0 auth_method 0 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e061640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e061640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.181+0000 7f521e061640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521e862640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521e862640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521e862640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521d860640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521d860640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521d860640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521e061640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521e061640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.177 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.182+0000 7f521e061640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.184+0000 7f521e862640 10 monclient(hunting): handle_auth_done global_id 15444 payload 824 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.184+0000 7f521e862640 1 -- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f52180b2df0 msgr2=0x7f5218144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.184+0000 7f521e862640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f52180b2df0 0x7f5218144560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.184+0000 7f521e862640 1 -- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52180b04a0 msgr2=0x7f5218138fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.184+0000 7f521e862640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52180b04a0 0x7f5218138fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.185+0000 7f521e862640 10 monclient: _finish_hunting 0 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.185+0000 7f521e862640 1 monclient: found mon.noname-a 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.185+0000 7f521e862640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.185+0000 7f521e862640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.83:3300/0 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.185+0000 7f521e862640 1 -- 172.21.15.83:0/1564584640 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521813a360 con 0x7f5218140b10 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.185+0000 7f521d860640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f52180b2df0 0x7f5218144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-25T02:35:14.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.187+0000 7f521e862640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5218140b10 0x7f5218144aa0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f5214033000 tx=0x7f52140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.187+0000 7f520effd640 10 client.?.objecter ms_handle_connect 0x7f5218140b10 2024-03-25T02:35:14.179 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.187+0000 7f520effd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:14.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 1 -- 172.21.15.83:0/1564584640 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214040070 con 0x7f5218140b10 2024-03-25T02:35:14.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T02:35:14.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: dump: 2024-03-25T02:35:14.180 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.180 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: _finish_auth 0 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 1 -- 172.21.15.83:0/1564584640 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5214039cf0 con 0x7f5218140b10 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 5 monclient: authenticate success, global_id 15444 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 10 monclient: _renew_subs 2024-03-25T02:35:14.181 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f521813a640 con 0x7f5218140b10 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 1 -- 172.21.15.83:0/1564584640 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5214035040 con 0x7f5218140b10 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f520effd640 10 monclient: dump: 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.182 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f521c85e640 4 set_mon_vals no callback set 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 10 client.15444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 10 monclient: _renew_subs 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5218146d70 con 0x7f5218140b10 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.190+0000 7f52202c5640 1 librados: init done 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.191+0000 7f52202c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.191+0000 7f52202c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.191+0000 7f52202c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.83:3300/0 2024-03-25T02:35:14.183 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.191+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5218075810 con 0x7f5218140b10 2024-03-25T02:35:14.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.211+0000 7f520effd640 1 -- 172.21.15.83:0/1564584640 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f521404b060 con 0x7f5218140b10 2024-03-25T02:35:14.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.211+0000 7f520effd640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f51fc06da70 0x7f51fc06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.211+0000 7f521e061640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f51fc06da70 0x7f51fc06ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.211+0000 7f521e061640 10 monclient: get_auth_request con 0x7f51fc06da70 auth_method 0 2024-03-25T02:35:14.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.212+0000 7f521e061640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f51fc06da70 0x7f51fc06ff30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f52100113e0 tx=0x7f521000b040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.212+0000 7f520effd640 10 client.15444.objecter ms_handle_connect 0x7f51fc06da70 2024-03-25T02:35:14.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.214+0000 7f520effd640 1 -- 172.21.15.83:0/1564584640 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f5214079040 con 0x7f5218140b10 2024-03-25T02:35:14.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.214+0000 7f520effd640 10 client.15444.objecter ms_dispatch 0x7f5218000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:14.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.214+0000 7f520effd640 3 client.15444.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:14.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.214+0000 7f520effd640 3 client.15444.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:14.203 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.214+0000 7f520effd640 20 client.15444.objecter dump_active .. 0 homeless 2024-03-25T02:35:14.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.454+0000 7f520effd640 1 -- 172.21.15.83:0/1564584640 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f521407afa0 con 0x7f5218140b10 2024-03-25T02:35:14.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.454+0000 7f520effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:14.443 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.454+0000 7f520effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:14.619 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.630+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f5218076d20 con 0x7f51fc06da70 2024-03-25T02:35:14.620 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.632+0000 7f520effd640 1 -- 172.21.15.83:0/1564584640 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5210017440 con 0x7f51fc06da70 2024-03-25T02:35:14.620 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 6 to deep-scrub 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 10 librados: watch_flush enter 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 10 librados: watch_flush exit 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 20 client.15444.objecter shutdown clearing up homeless session... 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 10 client.15444.objecter successfully canceled tick 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f51fc06da70 msgr2=0x7f51fc06ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f51fc06da70 0x7f51fc06ff30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f52100113e0 tx=0x7f521000b040 comp rx=0 tx=0).stop 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 10 monclient: shutdown 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5218140b10 msgr2=0x7f5218144aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5218140b10 0x7f5218144aa0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f5214033000 tx=0x7f52140046c0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 shutdown_connections 2024-03-25T02:35:14.623 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f51fc06da70 0x7f51fc06ff30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5218140b10 0x7f5218144aa0 unknown :-1 s=CLOSED pgs=3215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f52180b2df0 0x7f5218144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 --2- 172.21.15.83:0/1564584640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52180b04a0 0x7f5218138fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 >> 172.21.15.83:0/1564584640 conn(0x7f52180b8a00 msgr2=0x7f521813e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:14.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 shutdown_connections 2024-03-25T02:35:14.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 -- 172.21.15.83:0/1564584640 wait complete. 2024-03-25T02:35:14.624 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.633+0000 7f52202c5640 1 librados: shutdown 2024-03-25T02:35:14.633 INFO:tasks.ceph:Scrubbing osd.7 2024-03-25T02:35:14.633 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-25T02:35:14.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.759+0000 7f240693a640 10 monclient: get_monmap_and_config 2024-03-25T02:35:14.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.759+0000 7f240693a640 10 monclient: build_initial_monmap 2024-03-25T02:35:14.748 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.759+0000 7f240693a640 10 monclient: monmap: 2024-03-25T02:35:14.748 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:14.748 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:14.760817+0000 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:14.760817+0000 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:14.749 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 1 Processor -- start 2024-03-25T02:35:14.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 1 -- start start 2024-03-25T02:35:14.751 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient: init 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24000a9c10 0x7f24000a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient(hunting): picked mon.noname-b con 0x7f24000a9c10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24001403b0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2400143c90 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24000abe60 con 0x7f24001403b0 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24000ac140 con 0x7f2400143c90 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f24001430f0 con 0x7f24000a9c10 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.760+0000 7f240693a640 10 monclient(hunting): authenticate will time out at 3433.950754s 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f23fffff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24000a9c10 0x7f24000a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.752 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f23ff7fe640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7f24001403b0 0x7f2400143580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:60360/0 (socket says 172.21.15.83:60360) 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f2404ed7640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2400143c90 0x7f240014d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:38162/0 (socket says 172.21.15.83:38162) 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f23ff7fe640 1 -- 172.21.15.83:0/2295054425 learned_addr learned my addr 172.21.15.83:0/2295054425 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f23fffff640 10 monclient(hunting): get_auth_request con 0x7f24000a9c10 auth_method 0 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f23fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f23fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.761+0000 7f23fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.762+0000 7f23fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.762+0000 7f23fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.753 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.762+0000 7f23fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.763+0000 7f23feffd640 1 -- 172.21.15.83:0/2295054425 <== mon.1 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2524032414 0 0) 0x7f24000ac140 con 0x7f2400143c90 2024-03-25T02:35:14.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.763+0000 7f23feffd640 10 monclient(hunting): _negotiate 2024-03-25T02:35:14.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.763+0000 7f23feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.763+0000 7f23feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.763+0000 7f23feffd640 10 monclient(hunting): my global_id is 22099 2024-03-25T02:35:14.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23feffd640 1 -- 172.21.15.83:0/2295054425 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23e8003520 con 0x7f2400143c90 2024-03-25T02:35:14.754 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 10 monclient(hunting): handle_auth_done global_id 17282 payload 274 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 1 -- 172.21.15.83:0/2295054425 >> v1:172.21.15.153:6789/0 conn(0x7f2400143c90 legacy=0x7f240014d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 1 -- 172.21.15.83:0/2295054425 >> v1:172.21.15.83:6789/0 conn(0x7f24001403b0 legacy=0x7f2400143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 1 monclient: found mon.noname-b 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.764+0000 7f23fffff640 1 -- 172.21.15.83:0/2295054425 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24000b3460 con 0x7f24000a9c10 2024-03-25T02:35:14.755 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.766+0000 7f23fffff640 1 --2- 172.21.15.83:0/2295054425 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24000a9c10 0x7f24000a9ff0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f23f00097d0 tx=0x7f23f002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89574602dc101e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 1 -- 172.21.15.83:0/2295054425 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f0035020 con 0x7f24000a9c10 2024-03-25T02:35:14.759 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: dump: 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: _finish_auth 0 2024-03-25T02:35:14.760 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 1 -- 172.21.15.83:0/2295054425 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f0031800 con 0x7f24000a9c10 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 1 -- 172.21.15.83:0/2295054425 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f0031bd0 con 0x7f24000a9c10 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 5 monclient: authenticate success, global_id 17282 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 4 set_mon_vals no callback set 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f23feffd640 10 monclient: dump: 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.761 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 10 monclient: shutdown 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 1 -- 172.21.15.83:0/2295054425 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24000a9c10 msgr2=0x7f24000a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 1 --2- 172.21.15.83:0/2295054425 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24000a9c10 0x7f24000a9ff0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7f23f00097d0 tx=0x7f23f002b8a0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 1 -- 172.21.15.83:0/2295054425 shutdown_connections 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 1 --2- 172.21.15.83:0/2295054425 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f24000a9c10 0x7f24000a9ff0 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 1 -- 172.21.15.83:0/2295054425 >> 172.21.15.83:0/2295054425 conn(0x7f24000b8a60 msgr2=0x7f24000b8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:14.762 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 1 -- 172.21.15.83:0/2295054425 shutdown_connections 2024-03-25T02:35:14.763 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.770+0000 7f240693a640 1 -- 172.21.15.83:0/2295054425 wait complete. 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.771+0000 7f240693a640 10 monclient: build_initial_monmap 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.771+0000 7f240693a640 10 monclient: monmap: 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.766 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.771+0000 7f240693a640 1 librados: starting msgr at 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.771+0000 7f240693a640 1 librados: starting objecter 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.771+0000 7f240693a640 1 Processor -- start 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 -- start start 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 librados: setting wanted keys 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 librados: calling monclient init 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient: init 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24000a9c10 0x7f2400136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): picked mon.noname-a con 0x7f24000a9c10 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24001403b0 0x7f2400138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): picked mon.noname-b con 0x7f24001403b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2400143c90 0x7f2400134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): picked mon.noname-c con 0x7f2400143c90 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2400140990 con 0x7f24000a9c10 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2400140810 con 0x7f24001403b0 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f2400140b10 con 0x7f2400143c90 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.772+0000 7f240693a640 10 monclient(hunting): authenticate will time out at 3433.962695s 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24000a9c10 0x7f2400136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f2404ed7640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2400143c90 0x7f2400134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23ff7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24001403b0 0x7f2400138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23fffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24000a9c10 0x7f2400136b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33040/0 (socket says 172.21.15.83:33040) 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f2404ed7640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2400143c90 0x7f2400134d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:43318/0 (socket says 172.21.15.83:43318) 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23fffff640 1 -- 172.21.15.83:0/2078816716 learned_addr learned my addr 172.21.15.83:0/2078816716 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23fffff640 10 monclient(hunting): get_auth_request con 0x7f24000a9c10 auth_method 0 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f2404ed7640 10 monclient(hunting): get_auth_request con 0x7f2400143c90 auth_method 0 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f2404ed7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f2404ed7640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f2404ed7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23ff7fe640 10 monclient(hunting): get_auth_request con 0x7f24001403b0 auth_method 0 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.773+0000 7f23ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f2404ed7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f2404ed7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f2404ed7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f23fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f23fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f23fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f23ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f23ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.774+0000 7f23ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 10 monclient(hunting): handle_auth_done global_id 17285 payload 824 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 1 -- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24001403b0 msgr2=0x7f2400138bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24001403b0 0x7f2400138bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 1 -- 172.21.15.83:0/2078816716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24000a9c10 msgr2=0x7f2400136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24000a9c10 0x7f2400136b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 10 monclient: _finish_hunting 0 2024-03-25T02:35:14.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 1 monclient: found mon.noname-c 2024-03-25T02:35:14.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:14.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:14.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.776+0000 7f2404ed7640 1 -- 172.21.15.83:0/2078816716 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24001390f0 con 0x7f2400143c90 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.779+0000 7f2404ed7640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2400143c90 0x7f2400134d40 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f23f800c700 tx=0x7f23f800cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.779+0000 7f23fcff9640 10 client.?.objecter ms_handle_connect 0x7f2400143c90 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.779+0000 7f23fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f8017070 con 0x7f2400143c90 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f23fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f23fcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f23fcff9640 10 monclient: dump: 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.773 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f23fcff9640 10 monclient: _finish_auth 0 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f23fcff9640 10 monclient: _check_auth_tickets 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f23fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 5 monclient: authenticate success, global_id 17285 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 monclient: _renew_subs 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24001381d0 con 0x7f2400143c90 2024-03-25T02:35:14.774 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 client.17285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 monclient: _renew_subs 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2400135280 con 0x7f2400143c90 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 1 librados: init done 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 client.17285.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 15 client.17285.objecter _session_command_op_assign -1 1 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 20 client.17285.objecter _get_session osd=-1 returning homeless 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 client.17285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.781+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2400076d20 con 0x7f2400143c90 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.782+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f800ce10 con 0x7f2400143c90 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.782+0000 7f23fcff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.782+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f8021060 con 0x7f2400143c90 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.782+0000 7f23fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.775 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.782+0000 7f23fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.782+0000 7f23fcff9640 10 monclient: dump: 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.782+0000 7f23fe7fc640 4 set_mon_vals no callback set 2024-03-25T02:35:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.804+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f23f801a070 con 0x7f2400143c90 2024-03-25T02:35:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.804+0000 7f23fcff9640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f23ec06da20 0x7f23ec06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.804+0000 7f23fffff640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f23ec06da20 0x7f23ec06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.793 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.804+0000 7f23fffff640 10 monclient: get_auth_request con 0x7f23ec06da20 auth_method 0 2024-03-25T02:35:14.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.805+0000 7f23fffff640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f23ec06da20 0x7f23ec06fee0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f23f0000c00 tx=0x7f23f003c040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.805+0000 7f23fcff9640 10 client.17285.objecter ms_handle_connect 0x7f23ec06da20 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f23f8057400 con 0x7f2400143c90 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 10 client.17285.objecter ms_dispatch 0x7f2400000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 3 client.17285.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 3 client.17285.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 10 client.17285.objecter checking command 1 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7f23ec075720 0x7f23ec077bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 20 client.17285.objecter _get_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 20 client.17285.objecter put_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 15 client.17285.objecter _session_command_op_remove -1 1 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 20 client.17285.objecter _get_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 20 client.17285.objecter get_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 15 client.17285.objecter _session_command_op_assign 7 1 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 20 client.17285.objecter put_session s=0x7f23ec075050 osd=7 4 2024-03-25T02:35:14.796 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 10 client.17285.objecter _send_command 1 2024-03-25T02:35:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23f8014ec0 con 0x7f23ec075720 2024-03-25T02:35:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 20 client.17285.objecter dump_active .. 0 homeless 2024-03-25T02:35:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_get_version_reply(handle=1 version=1250) ==== 24+0+0 (secure 0 0 0) 0x7f23f8055600 con 0x7f2400143c90 2024-03-25T02:35:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.806+0000 7f23fcff9640 10 monclient: handle_get_version_reply finishing 1 version 1250 2024-03-25T02:35:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.807+0000 7f23ff7fe640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7f23ec075720 0x7f23ec077bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.797 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.807+0000 7f23ff7fe640 10 monclient: get_auth_request con 0x7f23ec075720 auth_method 0 2024-03-25T02:35:14.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.810+0000 7f23ff7fe640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7f23ec075720 0x7f23ec077bc0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.798 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.810+0000 7f23fcff9640 10 client.17285.objecter ms_handle_connect 0x7f23ec075720 2024-03-25T02:35:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.833+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== osd.7 v2:172.21.15.153:6826/2580901001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23f8014ec0 con 0x7f23ec075720 2024-03-25T02:35:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.833+0000 7f23fcff9640 10 client.17285.objecter ms_dispatch 0x7f2400000b90 command_reply(tid 1: 0 ) 2024-03-25T02:35:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.833+0000 7f23fcff9640 10 client.17285.objecter _finish_command 1 = system:0 2024-03-25T02:35:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.833+0000 7f23fcff9640 20 client.17285.objecter put_session s=0x7f23ec075050 osd=7 4 2024-03-25T02:35:14.822 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.833+0000 7f23fcff9640 15 client.17285.objecter _session_command_op_remove 7 1 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 10 client.17285.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 15 client.17285.objecter _session_command_op_assign -1 2 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 20 client.17285.objecter _get_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 20 client.17285.objecter put_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 20 client.17285.objecter _get_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 15 client.17285.objecter _session_command_op_remove -1 2 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 20 client.17285.objecter get_session s=0x7f23ec075050 osd=7 3 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 15 client.17285.objecter _session_command_op_assign 7 2 2024-03-25T02:35:14.841 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 20 client.17285.objecter put_session s=0x7f23ec075050 osd=7 4 2024-03-25T02:35:14.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 10 client.17285.objecter _send_command 2 2024-03-25T02:35:14.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.851+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 --> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2400075810 con 0x7f23ec075720 2024-03-25T02:35:14.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.865+0000 7f23fcff9640 1 -- 172.21.15.83:0/2078816716 <== osd.7 v2:172.21.15.153:6826/2580901001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2400075810 con 0x7f23ec075720 2024-03-25T02:35:14.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.865+0000 7f23fcff9640 10 client.17285.objecter ms_dispatch 0x7f2400000b90 command_reply(tid 2: 0 ) 2024-03-25T02:35:14.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.865+0000 7f23fcff9640 10 client.17285.objecter _finish_command 2 = system:0 2024-03-25T02:35:14.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.865+0000 7f23fcff9640 20 client.17285.objecter put_session s=0x7f23ec075050 osd=7 4 2024-03-25T02:35:14.854 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.865+0000 7f23fcff9640 15 client.17285.objecter _session_command_op_remove 7 2 2024-03-25T02:35:14.855 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-25T02:35:14.855 INFO:teuthology.orchestra.run.smithi083.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-25T02:35:14.855 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-25T02:35:14.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.865+0000 7f240693a640 10 librados: watch_flush enter 2024-03-25T02:35:14.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 10 librados: watch_flush exit 2024-03-25T02:35:14.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 10 client.17285.objecter close_session for osd.7 2024-03-25T02:35:14.856 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7f23ec075720 msgr2=0x7f23ec077bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6826/2580901001,v1:172.21.15.153:6827/2580901001] conn(0x7f23ec075720 0x7f23ec077bc0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 20 client.17285.objecter put_session s=0x7f23ec075050 osd=7 1 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 20 client.17285.objecter shutdown clearing up homeless session... 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 10 client.17285.objecter successfully canceled tick 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f23ec06da20 msgr2=0x7f23ec06fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f23ec06da20 0x7f23ec06fee0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f23f0000c00 tx=0x7f23f003c040 comp rx=0 tx=0).stop 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 10 monclient: shutdown 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2400143c90 msgr2=0x7f2400134d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 --2- 172.21.15.83:0/2078816716 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f2400143c90 0x7f2400134d40 secure :-1 s=READY pgs=3207 cs=0 l=1 rev1=1 crypto rx=0x7f23f800c700 tx=0x7f23f800cbc0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f2404ed7640 1 -- 172.21.15.83:0/2078816716 reap_dead start 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 shutdown_connections 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 >> 172.21.15.83:0/2078816716 conn(0x7f24000b8a60 msgr2=0x7f240013f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 shutdown_connections 2024-03-25T02:35:14.857 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 -- 172.21.15.83:0/2078816716 wait complete. 2024-03-25T02:35:14.858 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.866+0000 7f240693a640 1 librados: shutdown 2024-03-25T02:35:14.868 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-25T02:35:14.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.981+0000 7f687ef10640 10 monclient: get_monmap_and_config 2024-03-25T02:35:14.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.981+0000 7f687ef10640 10 monclient: build_initial_monmap 2024-03-25T02:35:14.970 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.981+0000 7f687ef10640 10 monclient: monmap: 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:14.982956+0000 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:14.982956+0000 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:14.971 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 1 Processor -- start 2024-03-25T02:35:14.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 1 -- start start 2024-03-25T02:35:14.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient: init 2024-03-25T02:35:14.973 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6878072660 0x7f687813ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient(hunting): picked mon.noname-c con 0x7f6878072660 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6878140500 0x7f687806d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient(hunting): picked mon.noname-a con 0x7f6878140500 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f687806daf0 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6878072a40 con 0x7f687806daf0 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f687806d190 con 0x7f6878140500 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f6878140c40 con 0x7f6878072660 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:14.974 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.982+0000 7f687ef10640 10 monclient(hunting): authenticate will time out at 3434.172815s 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f686ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6878140500 0x7f687806d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f687ccac640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6878072660 0x7f687813ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f686ffff640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f6878140500 0x7f687806d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33048/0 (socket says 172.21.15.83:33048) 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f686ffff640 1 -- 172.21.15.83:0/3190628891 learned_addr learned my addr 172.21.15.83:0/3190628891 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f686ffff640 10 monclient(hunting): get_auth_request con 0x7f6878140500 auth_method 0 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f686ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f686ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f686ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.983+0000 7f687ccac640 10 monclient(hunting): get_auth_request con 0x7f6878072660 auth_method 0 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f687ccac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f687ccac640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f687ccac640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f687ccac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f687ccac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.975 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f687ccac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f686ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f686ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.984+0000 7f686ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 10 monclient(hunting): handle_auth_done global_id 22105 payload 274 2024-03-25T02:35:14.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 1 -- 172.21.15.83:0/3190628891 >> v2:172.21.15.83:3300/0 conn(0x7f6878140500 msgr2=0x7f687806d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:14.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 1 --2- 172.21.15.83:0/3190628891 >> v2:172.21.15.83:3300/0 conn(0x7f6878140500 0x7f687806d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.976 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 1 -- 172.21.15.83:0/3190628891 >> v1:172.21.15.83:6789/0 conn(0x7f687806daf0 legacy=0x7f6878146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 10 monclient: _finish_hunting 0 2024-03-25T02:35:14.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 1 monclient: found mon.noname-c 2024-03-25T02:35:14.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:14.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3300/0 2024-03-25T02:35:14.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.986+0000 7f687ccac640 1 -- 172.21.15.83:0/3190628891 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6878071ea0 con 0x7f6878072660 2024-03-25T02:35:14.977 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.989+0000 7f687ccac640 1 --2- 172.21.15.83:0/3190628891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6878072660 0x7f687813ffc0 secure :-1 s=READY pgs=5549 cs=0 l=1 rev1=1 crypto rx=0x7f68780b4580 tx=0x7f687402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9456b9a34212b777 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.980 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 1 -- 172.21.15.83:0/3190628891 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68740315a0 con 0x7f6878072660 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: dump: 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.981 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: _finish_auth 0 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: _check_auth_tickets 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f687ef10640 5 monclient: authenticate success, global_id 22105 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 1 -- 172.21.15.83:0/3190628891 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6874037070 con 0x7f6878072660 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f687ef10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f687ef10640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f687ef10640 4 set_mon_vals no callback set 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 1 -- 172.21.15.83:0/3190628891 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874031960 con 0x7f6878072660 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.991+0000 7f686f7fe640 10 monclient: dump: 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.982 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.983 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 10 monclient: shutdown 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 -- 172.21.15.83:0/3190628891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6878072660 msgr2=0x7f687813ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 --2- 172.21.15.83:0/3190628891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6878072660 0x7f687813ffc0 secure :-1 s=READY pgs=5549 cs=0 l=1 rev1=1 crypto rx=0x7f68780b4580 tx=0x7f687402b840 comp rx=0 tx=0).stop 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 -- 172.21.15.83:0/3190628891 shutdown_connections 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 --2- 172.21.15.83:0/3190628891 >> v2:172.21.15.83:3300/0 conn(0x7f6878140500 0x7f687806d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 --2- 172.21.15.83:0/3190628891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6878072660 0x7f687813ffc0 unknown :-1 s=CLOSED pgs=5549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 -- 172.21.15.83:0/3190628891 >> 172.21.15.83:0/3190628891 conn(0x7f6878067e90 msgr2=0x7f68780b69c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 -- 172.21.15.83:0/3190628891 shutdown_connections 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.992+0000 7f687ef10640 1 -- 172.21.15.83:0/3190628891 wait complete. 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.993+0000 7f687ef10640 10 monclient: build_initial_monmap 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.993+0000 7f687ef10640 10 monclient: monmap: 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:14.987 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 librados: starting msgr at 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 librados: starting objecter 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 Processor -- start 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 -- start start 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 librados: setting wanted keys 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 librados: calling monclient init 2024-03-25T02:35:14.988 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient: init 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687806daf0 0x7f687811ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): picked mon.noname-b con 0x7f687806daf0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6878072660 0x7f6878146250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): picked mon.noname-a con 0x7f6878072660 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6878140500 0x7f68781d88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): picked mon.noname-c con 0x7f6878140500 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f687806e0c0 con 0x7f6878072660 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f687806df40 con 0x7f687806daf0 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f68780b0e80 con 0x7f6878140500 2024-03-25T02:35:14.989 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.994+0000 7f687ef10640 10 monclient(hunting): authenticate will time out at 3434.184904s 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f686ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6878072660 0x7f6878146250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687ccac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687806daf0 0x7f687811ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687d4ad640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6878140500 0x7f68781d88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687ccac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687806daf0 0x7f687811ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:46048/0 (socket says 172.21.15.83:46048) 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687ccac640 1 -- 172.21.15.83:0/1407041210 learned_addr learned my addr 172.21.15.83:0/1407041210 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687ccac640 10 monclient(hunting): get_auth_request con 0x7f687806daf0 auth_method 0 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687ccac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687ccac640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f687ccac640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f686ffff640 10 monclient(hunting): get_auth_request con 0x7f6878072660 auth_method 0 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f686ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f686ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.990 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.995+0000 7f686ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687d4ad640 10 monclient(hunting): get_auth_request con 0x7f6878140500 auth_method 0 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687d4ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687d4ad640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687d4ad640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687ccac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687ccac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687ccac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687d4ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.991 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687d4ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f687d4ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f686ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:14.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f686ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:14.992 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.996+0000 7f686ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:14.994 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 10 monclient(hunting): handle_auth_done global_id 22108 payload 824 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 1 -- 172.21.15.83:0/1407041210 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6878140500 msgr2=0x7f68781d88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6878140500 0x7f68781d88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 1 -- 172.21.15.83:0/1407041210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6878072660 msgr2=0x7f6878146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6878072660 0x7f6878146250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 10 monclient: _finish_hunting 0 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 1 monclient: found mon.noname-b 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:14.999+0000 7f687ccac640 1 -- 172.21.15.83:0/1407041210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f687811efa0 con 0x7f687806daf0 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.001+0000 7f687ccac640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687806daf0 0x7f687811ea60 secure :-1 s=READY pgs=5550 cs=0 l=1 rev1=1 crypto rx=0x7f6874009890 tx=0x7f68740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.001+0000 7f686d7fa640 10 client.?.objecter ms_handle_connect 0x7f687806daf0 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.001+0000 7f686d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.004+0000 7f686d7fa640 1 -- 172.21.15.83:0/1407041210 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874032630 con 0x7f687806daf0 2024-03-25T02:35:14.995 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: dump: 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: _finish_auth 0 2024-03-25T02:35:14.996 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: _check_auth_tickets 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 5 monclient: authenticate success, global_id 22108 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 1 -- 172.21.15.83:0/1407041210 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68740327d0 con 0x7f687806daf0 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 1 -- 172.21.15.83:0/1407041210 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874032bc0 con 0x7f687806daf0 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 monclient: _renew_subs 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68781435b0 con 0x7f687806daf0 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686d7fa640 10 monclient: dump: 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.997 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f686effd640 4 set_mon_vals no callback set 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 client.22108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 monclient: _renew_subs 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68781d6050 con 0x7f687806daf0 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 1 librados: init done 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:14.998 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:14.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:14.999 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.005+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6878075810 con 0x7f687806daf0 2024-03-25T02:35:15.245 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.257+0000 7f686d7fa640 1 -- 172.21.15.83:0/1407041210 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f687404e430 con 0x7f687806daf0 2024-03-25T02:35:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.257+0000 7f686d7fa640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f686406da70 0x7f686406ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.257+0000 7f686ffff640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f686406da70 0x7f686406ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:15.246 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.257+0000 7f686ffff640 10 monclient: get_auth_request con 0x7f686406da70 auth_method 0 2024-03-25T02:35:15.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.257+0000 7f686d7fa640 10 client.22108.objecter ms_handle_connect 0x7f686406da70 2024-03-25T02:35:15.247 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.257+0000 7f686ffff640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f686406da70 0x7f686406ff30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f68780b5840 tx=0x7f6868009240 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.260+0000 7f686d7fa640 1 -- 172.21.15.83:0/1407041210 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f687402f070 con 0x7f687806daf0 2024-03-25T02:35:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.260+0000 7f686d7fa640 10 client.22108.objecter ms_dispatch 0x7f6878000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.260+0000 7f686d7fa640 3 client.22108.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.260+0000 7f686d7fa640 3 client.22108.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:15.249 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.260+0000 7f686d7fa640 20 client.22108.objecter dump_active .. 0 homeless 2024-03-25T02:35:15.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.272+0000 7f686d7fa640 1 -- 172.21.15.83:0/1407041210 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6874040070 con 0x7f687806daf0 2024-03-25T02:35:15.260 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.272+0000 7f686d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:15.261 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.272+0000 7f686d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:15.436 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.447+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f6878076d20 con 0x7f686406da70 2024-03-25T02:35:15.437 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.448+0000 7f686d7fa640 1 -- 172.21.15.83:0/1407041210 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6868011a30 con 0x7f686406da70 2024-03-25T02:35:15.437 INFO:teuthology.orchestra.run.smithi083.stderr:instructed osd(s) 7 to deep-scrub 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 10 librados: watch_flush enter 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 10 librados: watch_flush exit 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 20 client.22108.objecter shutdown clearing up homeless session... 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 10 client.22108.objecter successfully canceled tick 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f686406da70 msgr2=0x7f686406ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f686406da70 0x7f686406ff30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f68780b5840 tx=0x7f6868009240 comp rx=0 tx=0).stop 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 10 monclient: shutdown 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687806daf0 msgr2=0x7f687811ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:15.440 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687806daf0 0x7f687811ea60 secure :-1 s=READY pgs=5550 cs=0 l=1 rev1=1 crypto rx=0x7f6874009890 tx=0x7f68740046c0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 shutdown_connections 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f686406da70 0x7f686406ff30 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f6878140500 0x7f68781d88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6878072660 0x7f6878146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 --2- 172.21.15.83:0/1407041210 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f687806daf0 0x7f687811ea60 unknown :-1 s=CLOSED pgs=5550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 >> 172.21.15.83:0/1407041210 conn(0x7f6878067e90 msgr2=0x7f68780b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 shutdown_connections 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 -- 172.21.15.83:0/1407041210 wait complete. 2024-03-25T02:35:15.441 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.451+0000 7f687ef10640 1 librados: shutdown 2024-03-25T02:35:15.450 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-25T02:35:15.554 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.566+0000 7f47ce17a640 10 monclient: get_monmap_and_config 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.566+0000 7f47ce17a640 10 monclient: build_initial_monmap 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.566+0000 7f47ce17a640 10 monclient: monmap: 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:15.567124+0000 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:15.567124+0000 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:15.555 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:15.556 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:15.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.566+0000 7f47ce17a640 1 Processor -- start 2024-03-25T02:35:15.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.566+0000 7f47ce17a640 1 -- start start 2024-03-25T02:35:15.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.566+0000 7f47ce17a640 10 monclient: init 2024-03-25T02:35:15.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:15.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-03-25T02:35:15.558 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 1 --2- >> v2:172.21.15.83:3300/0 conn(0x7f47c80b04a0 0x7f47c8146c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient(hunting): picked mon.noname-a con 0x7f47c80b04a0 addr v2:172.21.15.83:3300/0 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47c80b2df0 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c8140b10 0x7f47c8147180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient(hunting): picked mon.noname-b con 0x7f47c8140b10 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47c80b2410 con 0x7f47c80b2df0 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 1 -- --> v2:172.21.15.83:3300/0 -- mon_getmap magic: 0 -- 0x7f47c80b26f0 con 0x7f47c80b04a0 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f47c80b29d0 con 0x7f47c8140b10 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47ce17a640 10 monclient(hunting): authenticate will time out at 3434.757087s 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47c7fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c8140b10 0x7f47c8147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:15.559 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47beffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f47c80b2df0 0x7f47c8140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.83:38172/0 (socket says 172.21.15.83:38172) 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47beffd640 1 -- 172.21.15.83:0/2209042587 learned_addr learned my addr 172.21.15.83:0/2209042587 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.567+0000 7f47c77fe640 1 --2- 172.21.15.83:0/2209042587 >> v2:172.21.15.83:3300/0 conn(0x7f47c80b04a0 0x7f47c8146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c7fff640 10 monclient(hunting): get_auth_request con 0x7f47c8140b10 auth_method 0 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c77fe640 10 monclient(hunting): get_auth_request con 0x7f47c80b04a0 auth_method 0 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:15.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:15.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:15.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:15.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.568+0000 7f47c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:15.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 10 monclient(hunting): handle_auth_done global_id 17291 payload 274 2024-03-25T02:35:15.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 1 -- 172.21.15.83:0/2209042587 >> v2:172.21.15.83:3300/0 conn(0x7f47c80b04a0 msgr2=0x7f47c8146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 1 --2- 172.21.15.83:0/2209042587 >> v2:172.21.15.83:3300/0 conn(0x7f47c80b04a0 0x7f47c8146c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 1 -- 172.21.15.83:0/2209042587 >> v1:172.21.15.153:6789/0 conn(0x7f47c80b2df0 legacy=0x7f47c8140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 1 monclient: found mon.noname-b 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.570+0000 7f47c7fff640 1 -- 172.21.15.83:0/2209042587 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c80b3380 con 0x7f47c8140b10 2024-03-25T02:35:15.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.572+0000 7f47c7fff640 1 --2- 172.21.15.83:0/2209042587 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c8140b10 0x7f47c8147180 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009dc0 tx=0x7f47c002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298a5661cc318a97 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:15.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 1 -- 172.21.15.83:0/2209042587 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0039070 con 0x7f47c8140b10 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: dump: 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:15.565 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: _finish_auth 0 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 1 -- 172.21.15.83:0/2209042587 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c002ccd0 con 0x7f47c8140b10 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 1 -- 172.21.15.83:0/2209042587 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0034660 con 0x7f47c8140b10 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47c6ffd640 10 monclient: dump: 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:15.566 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:15.567 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:15.567 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:15.567 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:15.567 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:15.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47ce17a640 5 monclient: authenticate success, global_id 17291 2024-03-25T02:35:15.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47ce17a640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:15.567 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.575+0000 7f47ce17a640 4 set_mon_vals no callback set 2024-03-25T02:35:15.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 10 monclient: shutdown 2024-03-25T02:35:15.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:15.570 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 -- 172.21.15.83:0/2209042587 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c8140b10 msgr2=0x7f47c8147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 --2- 172.21.15.83:0/2209042587 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c8140b10 0x7f47c8147180 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009dc0 tx=0x7f47c002bf20 comp rx=0 tx=0).stop 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 -- 172.21.15.83:0/2209042587 shutdown_connections 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 --2- 172.21.15.83:0/2209042587 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c8140b10 0x7f47c8147180 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 --2- 172.21.15.83:0/2209042587 >> v2:172.21.15.83:3300/0 conn(0x7f47c80b04a0 0x7f47c8146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 -- 172.21.15.83:0/2209042587 >> 172.21.15.83:0/2209042587 conn(0x7f47c80b8a00 msgr2=0x7f47c80b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 -- 172.21.15.83:0/2209042587 shutdown_connections 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.576+0000 7f47ce17a640 1 -- 172.21.15.83:0/2209042587 wait complete. 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.577+0000 7f47ce17a640 10 monclient: build_initial_monmap 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.577+0000 7f47ce17a640 10 monclient: monmap: 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:15.571 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.577+0000 7f47ce17a640 1 librados: starting msgr at 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.577+0000 7f47ce17a640 1 librados: starting objecter 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.577+0000 7f47ce17a640 1 Processor -- start 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 -- start start 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 librados: setting wanted keys 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 librados: calling monclient init 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient: init 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:15.572 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c80b04a0 0x7f47c81470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): picked mon.noname-c con 0x7f47c80b04a0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47c80b2df0 0x7f47c8139340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): picked mon.noname-a con 0x7f47c80b2df0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8140b10 0x7f47c8134ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): picked mon.noname-b con 0x7f47c8140b10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f47c814dc70 con 0x7f47c80b2df0 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f47c814daf0 con 0x7f47c8140b10 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7f47c814ddf0 con 0x7f47c80b04a0 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.578+0000 7f47ce17a640 10 monclient(hunting): authenticate will time out at 3434.768592s 2024-03-25T02:35:15.573 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c77fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c80b04a0 0x7f47c81470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47beffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47c80b2df0 0x7f47c8139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8140b10 0x7f47c8134ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8140b10 0x7f47c8134ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.83:46056/0 (socket says 172.21.15.83:46056) 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c77fe640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c80b04a0 0x7f47c81470d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:43338/0 (socket says 172.21.15.83:43338) 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c7fff640 1 -- 172.21.15.83:0/722170846 learned_addr learned my addr 172.21.15.83:0/722170846 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c7fff640 10 monclient(hunting): get_auth_request con 0x7f47c8140b10 auth_method 0 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47beffd640 10 monclient(hunting): get_auth_request con 0x7f47c80b2df0 auth_method 0 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:15.574 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c77fe640 10 monclient(hunting): get_auth_request con 0x7f47c80b04a0 auth_method 0 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.579+0000 7f47c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:15.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.580+0000 7f47beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:15.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 10 monclient(hunting): handle_auth_done global_id 22114 payload 824 2024-03-25T02:35:15.576 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 1 -- 172.21.15.83:0/722170846 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c80b04a0 msgr2=0x7f47c81470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c80b04a0 0x7f47c81470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 1 -- 172.21.15.83:0/722170846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47c80b2df0 msgr2=0x7f47c8139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47c80b2df0 0x7f47c8139340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 1 monclient: found mon.noname-b 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3300/0 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.582+0000 7f47c7fff640 1 -- 172.21.15.83:0/722170846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c813a570 con 0x7f47c8140b10 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.585+0000 7f47c7fff640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8140b10 0x7f47c8134ad0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 crypto rx=0x7f47c002c530 tx=0x7f47c0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.585+0000 7f47c4ff9640 10 client.?.objecter ms_handle_connect 0x7f47c8140b10 2024-03-25T02:35:15.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.585+0000 7f47c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 1 -- 172.21.15.83:0/722170846 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0042020 con 0x7f47c8140b10 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: dump: 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:15.579 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: _finish_auth 0 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 1 -- 172.21.15.83:0/722170846 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c003a040 con 0x7f47c8140b10 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 5 monclient: authenticate success, global_id 22114 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 1 -- 172.21.15.83:0/722170846 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c0039070 con 0x7f47c8140b10 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 10 monclient: _renew_subs 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c813a850 con 0x7f47c8140b10 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c67fc640 4 set_mon_vals no callback set 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T02:35:15.580 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47c4ff9640 10 monclient: dump: 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 10 client.22114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 10 monclient: _renew_subs 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c81351f0 con 0x7f47c8140b10 2024-03-25T02:35:15.581 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.588+0000 7f47ce17a640 1 librados: init done 2024-03-25T02:35:15.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.589+0000 7f47ce17a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:15.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.589+0000 7f47ce17a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:15.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.589+0000 7f47ce17a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.153:3300/0 2024-03-25T02:35:15.582 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.589+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c8075810 con 0x7f47c8140b10 2024-03-25T02:35:15.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.607+0000 7f47c4ff9640 1 -- 172.21.15.83:0/722170846 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7f47c0045020 con 0x7f47c8140b10 2024-03-25T02:35:15.596 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.607+0000 7f47c4ff9640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f47a806da20 0x7f47a806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:15.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.607+0000 7f47c77fe640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f47a806da20 0x7f47a806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:15.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.607+0000 7f47c77fe640 10 monclient: get_auth_request con 0x7f47a806da20 auth_method 0 2024-03-25T02:35:15.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.608+0000 7f47c77fe640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f47a806da20 0x7f47a806fee0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009b40 tx=0x7f47b8009190 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:15.597 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.608+0000 7f47c4ff9640 10 client.22114.objecter ms_handle_connect 0x7f47a806da20 2024-03-25T02:35:15.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.610+0000 7f47c4ff9640 1 -- 172.21.15.83:0/722170846 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7f47c00799a0 con 0x7f47c8140b10 2024-03-25T02:35:15.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.610+0000 7f47c4ff9640 10 client.22114.objecter ms_dispatch 0x7f47c8000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:15.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.610+0000 7f47c4ff9640 3 client.22114.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:15.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.610+0000 7f47c4ff9640 3 client.22114.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:15.599 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.610+0000 7f47c4ff9640 20 client.22114.objecter dump_active .. 0 homeless 2024-03-25T02:35:15.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.861+0000 7f47c4ff9640 1 -- 172.21.15.83:0/722170846 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f47c004c030 con 0x7f47c8140b10 2024-03-25T02:35:15.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.861+0000 7f47c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:15.849 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:15.861+0000 7f47c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:16.049 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.061+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f47c80b7b80 con 0x7f47a806da20 2024-03-25T02:35:16.054 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.066+0000 7f47c4ff9640 1 -- 172.21.15.83:0/722170846 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+115921 (secure 0 0 0) 0x7f47c80b7b80 con 0x7f47a806da20 2024-03-25T02:35:16.055 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:35:16.056 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 10 librados: watch_flush enter 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 10 librados: watch_flush exit 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 20 client.22114.objecter shutdown clearing up homeless session... 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 10 client.22114.objecter successfully canceled tick 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f47a806da20 msgr2=0x7f47a806fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f47a806da20 0x7f47a806fee0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009b40 tx=0x7f47b8009190 comp rx=0 tx=0).stop 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 10 monclient: shutdown 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8140b10 msgr2=0x7f47c8134ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8140b10 0x7f47c8134ad0 secure :-1 s=READY pgs=5552 cs=0 l=1 rev1=1 crypto rx=0x7f47c002c530 tx=0x7f47c0004a40 comp rx=0 tx=0).stop 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 shutdown_connections 2024-03-25T02:35:16.058 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7f47a806da20 0x7f47a806fee0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f47c8140b10 0x7f47c8134ad0 unknown :-1 s=CLOSED pgs=5552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47c80b2df0 0x7f47c8139340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 --2- 172.21.15.83:0/722170846 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7f47c80b04a0 0x7f47c81470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.068+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 >> 172.21.15.83:0/722170846 conn(0x7f47c80b8a00 msgr2=0x7f47c813d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.069+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 shutdown_connections 2024-03-25T02:35:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.069+0000 7f47ce17a640 1 -- 172.21.15.83:0/722170846 wait complete. 2024-03-25T02:35:16.059 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:16.069+0000 7f47ce17a640 1 librados: shutdown 2024-03-25T02:35:16.068 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":2304,"stamp":"2024-03-25T02:35:14.933144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":89,"num_write":121,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":559900,"kb_used_data":30008,"kb_used_omap":12,"kb_used_meta":529779,"kb_avail":754414820,"statfs":{"total":773094113280,"available":772520775680,"internally_reserved":0,"allocated":30728192,"data_stored":13108610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":542494256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.953974"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2405,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.287136+0000","last_change":"2024-03-25T02:01:59.223551+0000","last_active":"2024-03-25T02:35:04.287136+0000","last_peered":"2024-03-25T02:35:04.287136+0000","last_clean":"2024-03-25T02:35:04.287136+0000","last_became_active":"2024-03-25T02:01:59.217800+0000","last_became_peered":"2024-03-25T02:01:59.217800+0000","last_unstale":"2024-03-25T02:35:04.287136+0000","last_undegraded":"2024-03-25T02:35:04.287136+0000","last_fullsized":"2024-03-25T02:35:04.287136+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:52:12.956708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2416,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.264084+0000","last_change":"2024-03-25T02:01:45.802865+0000","last_active":"2024-03-25T02:35:04.264084+0000","last_peered":"2024-03-25T02:35:04.264084+0000","last_clean":"2024-03-25T02:35:04.264084+0000","last_became_active":"2024-03-25T02:01:45.737540+0000","last_became_peered":"2024-03-25T02:01:45.737540+0000","last_unstale":"2024-03-25T02:35:04.264084+0000","last_undegraded":"2024-03-25T02:35:04.264084+0000","last_fullsized":"2024-03-25T02:35:04.264084+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:53:36.729760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2413,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.997516+0000","last_change":"2024-03-25T02:01:57.033633+0000","last_active":"2024-03-25T02:35:03.997516+0000","last_peered":"2024-03-25T02:35:03.997516+0000","last_clean":"2024-03-25T02:35:03.997516+0000","last_became_active":"2024-03-25T02:01:57.014842+0000","last_became_peered":"2024-03-25T02:01:57.014842+0000","last_unstale":"2024-03-25T02:35:03.997516+0000","last_undegraded":"2024-03-25T02:35:03.997516+0000","last_fullsized":"2024-03-25T02:35:03.997516+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:30:09.539808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2409,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.270310+0000","last_change":"2024-03-25T02:01:39.501653+0000","last_active":"2024-03-25T02:35:04.270310+0000","last_peered":"2024-03-25T02:35:04.270310+0000","last_clean":"2024-03-25T02:35:04.270310+0000","last_became_active":"2024-03-25T02:01:39.486234+0000","last_became_peered":"2024-03-25T02:01:39.486234+0000","last_unstale":"2024-03-25T02:35:04.270310+0000","last_undegraded":"2024-03-25T02:35:04.270310+0000","last_fullsized":"2024-03-25T02:35:04.270310+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:45:00.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2414,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.281889+0000","last_change":"2024-03-25T02:01:39.797384+0000","last_active":"2024-03-25T02:35:04.281889+0000","last_peered":"2024-03-25T02:35:04.281889+0000","last_clean":"2024-03-25T02:35:04.281889+0000","last_became_active":"2024-03-25T02:01:39.792053+0000","last_became_peered":"2024-03-25T02:01:39.792053+0000","last_unstale":"2024-03-25T02:35:04.281889+0000","last_undegraded":"2024-03-25T02:35:04.281889+0000","last_fullsized":"2024-03-25T02:35:04.281889+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:24:38.937942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2413,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.954110+0000","last_change":"2024-03-25T02:01:42.883607+0000","last_active":"2024-03-25T02:35:03.954110+0000","last_peered":"2024-03-25T02:35:03.954110+0000","last_clean":"2024-03-25T02:35:03.954110+0000","last_became_active":"2024-03-25T02:01:42.877188+0000","last_became_peered":"2024-03-25T02:01:42.877188+0000","last_unstale":"2024-03-25T02:35:03.954110+0000","last_undegraded":"2024-03-25T02:35:03.954110+0000","last_fullsized":"2024-03-25T02:35:03.954110+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:55:25.547629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.297649+0000","last_change":"2024-03-25T02:01:45.794313+0000","last_active":"2024-03-25T02:35:04.297649+0000","last_peered":"2024-03-25T02:35:04.297649+0000","last_clean":"2024-03-25T02:35:04.297649+0000","last_became_active":"2024-03-25T02:01:45.785913+0000","last_became_peered":"2024-03-25T02:01:45.785913+0000","last_unstale":"2024-03-25T02:35:04.297649+0000","last_undegraded":"2024-03-25T02:35:04.297649+0000","last_fullsized":"2024-03-25T02:35:04.297649+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:46:04.871320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.284677+0000","last_change":"2024-03-25T02:01:47.960650+0000","last_active":"2024-03-25T02:35:04.284677+0000","last_peered":"2024-03-25T02:35:04.284677+0000","last_clean":"2024-03-25T02:35:04.284677+0000","last_became_active":"2024-03-25T02:01:47.939684+0000","last_became_peered":"2024-03-25T02:01:47.939684+0000","last_unstale":"2024-03-25T02:35:04.284677+0000","last_undegraded":"2024-03-25T02:35:04.284677+0000","last_fullsized":"2024-03-25T02:35:04.284677+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:51:58.276672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"39'7","reported_seq":2427,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.926128+0000","last_change":"2024-03-25T02:01:58.313196+0000","last_active":"2024-03-25T02:35:03.926128+0000","last_peered":"2024-03-25T02:35:03.926128+0000","last_clean":"2024-03-25T02:35:03.926128+0000","last_became_active":"2024-03-25T02:01:58.306972+0000","last_became_peered":"2024-03-25T02:01:58.306972+0000","last_unstale":"2024-03-25T02:35:03.926128+0000","last_undegraded":"2024-03-25T02:35:03.926128+0000","last_fullsized":"2024-03-25T02:35:03.926128+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:05:07.200253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2414,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.292694+0000","last_change":"2024-03-25T02:01:44.749489+0000","last_active":"2024-03-25T02:35:04.292694+0000","last_peered":"2024-03-25T02:35:04.292694+0000","last_clean":"2024-03-25T02:35:04.292694+0000","last_became_active":"2024-03-25T02:01:44.731330+0000","last_became_peered":"2024-03-25T02:01:44.731330+0000","last_unstale":"2024-03-25T02:35:04.292694+0000","last_undegraded":"2024-03-25T02:35:04.292694+0000","last_fullsized":"2024-03-25T02:35:04.292694+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:40:48.117174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2455,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:09.893335+0000","last_change":"2024-03-25T02:35:09.893335+0000","last_active":"2024-03-25T02:35:09.893335+0000","last_peered":"2024-03-25T02:35:09.893335+0000","last_clean":"2024-03-25T02:35:09.893335+0000","last_became_active":"2024-03-25T02:01:38.008862+0000","last_became_peered":"2024-03-25T02:01:38.008862+0000","last_unstale":"2024-03-25T02:35:09.893335+0000","last_undegraded":"2024-03-25T02:35:09.893335+0000","last_fullsized":"2024-03-25T02:35:09.893335+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:09.890709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:09.890709+0000","last_clean_scrub_stamp":"2024-03-25T02:35:09.890709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:39:18.872661+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2408,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.307209+0000","last_change":"2024-03-25T02:01:55.459478+0000","last_active":"2024-03-25T02:35:04.307209+0000","last_peered":"2024-03-25T02:35:04.307209+0000","last_clean":"2024-03-25T02:35:04.307209+0000","last_became_active":"2024-03-25T02:01:55.452605+0000","last_became_peered":"2024-03-25T02:01:55.452605+0000","last_unstale":"2024-03-25T02:35:04.307209+0000","last_undegraded":"2024-03-25T02:35:04.307209+0000","last_fullsized":"2024-03-25T02:35:04.307209+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:52:09.879541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"23'77","reported_seq":2514,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.260520+0000","last_change":"2024-03-25T02:00:26.482050+0000","last_active":"2024-03-25T02:35:04.260520+0000","last_peered":"2024-03-25T02:35:04.260520+0000","last_clean":"2024-03-25T02:35:04.260520+0000","last_became_active":"2024-03-25T02:00:25.840109+0000","last_became_peered":"2024-03-25T02:00:25.840109+0000","last_unstale":"2024-03-25T02:35:04.260520+0000","last_undegraded":"2024-03-25T02:35:04.260520+0000","last_fullsized":"2024-03-25T02:35:04.260520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:15.236797+0000","last_clean_scrub_stamp":"2024-03-25T02:00:15.236797+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:57:04.634251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.149055+0000","last_change":"2024-03-25T02:00:33.873073+0000","last_active":"2024-03-25T02:35:04.149055+0000","last_peered":"2024-03-25T02:35:04.149055+0000","last_clean":"2024-03-25T02:35:04.149055+0000","last_became_active":"2024-03-25T02:00:33.865870+0000","last_became_peered":"2024-03-25T02:00:33.865870+0000","last_unstale":"2024-03-25T02:35:04.149055+0000","last_undegraded":"2024-03-25T02:35:04.149055+0000","last_fullsized":"2024-03-25T02:35:04.149055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2413,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.289586+0000","last_change":"2024-03-25T02:01:42.911218+0000","last_active":"2024-03-25T02:35:04.289586+0000","last_peered":"2024-03-25T02:35:04.289586+0000","last_clean":"2024-03-25T02:35:04.289586+0000","last_became_active":"2024-03-25T02:01:42.904032+0000","last_became_peered":"2024-03-25T02:01:42.904032+0000","last_unstale":"2024-03-25T02:35:04.289586+0000","last_undegraded":"2024-03-25T02:35:04.289586+0000","last_fullsized":"2024-03-25T02:35:04.289586+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:28:21.509321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2427,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:12.561114+0000","last_change":"2024-03-25T02:35:12.561880+0000","last_active":"2024-03-25T02:35:12.561114+0000","last_peered":"2024-03-25T02:35:12.561114+0000","last_clean":"2024-03-25T02:35:12.561114+0000","last_became_active":"2024-03-25T02:00:33.356691+0000","last_became_peered":"2024-03-25T02:00:33.356691+0000","last_unstale":"2024-03-25T02:35:12.561114+0000","last_undegraded":"2024-03-25T02:35:12.561114+0000","last_fullsized":"2024-03-25T02:35:12.561114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:12.559543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:12.559543+0000","last_clean_scrub_stamp":"2024-03-25T02:35:12.559543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T14:13:30.640010+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2413,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.274204+0000","last_change":"2024-03-25T02:01:39.128793+0000","last_active":"2024-03-25T02:35:04.274204+0000","last_peered":"2024-03-25T02:35:04.274204+0000","last_clean":"2024-03-25T02:35:04.274204+0000","last_became_active":"2024-03-25T02:01:39.122271+0000","last_became_peered":"2024-03-25T02:01:39.122271+0000","last_unstale":"2024-03-25T02:35:04.274204+0000","last_undegraded":"2024-03-25T02:35:04.274204+0000","last_fullsized":"2024-03-25T02:35:04.274204+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:57:32.356420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2422,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.928927+0000","last_change":"2024-03-25T02:00:33.459985+0000","last_active":"2024-03-25T02:35:03.928927+0000","last_peered":"2024-03-25T02:35:03.928927+0000","last_clean":"2024-03-25T02:35:03.928927+0000","last_became_active":"2024-03-25T02:00:33.429925+0000","last_became_peered":"2024-03-25T02:00:33.429925+0000","last_unstale":"2024-03-25T02:35:03.928927+0000","last_undegraded":"2024-03-25T02:35:03.928927+0000","last_fullsized":"2024-03-25T02:35:03.928927+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:09:49.004802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2409,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.268612+0000","last_change":"2024-03-25T02:01:53.314119+0000","last_active":"2024-03-25T02:35:04.268612+0000","last_peered":"2024-03-25T02:35:04.268612+0000","last_clean":"2024-03-25T02:35:04.268612+0000","last_became_active":"2024-03-25T02:01:53.306467+0000","last_became_peered":"2024-03-25T02:01:53.306467+0000","last_unstale":"2024-03-25T02:35:04.268612+0000","last_undegraded":"2024-03-25T02:35:04.268612+0000","last_fullsized":"2024-03-25T02:35:04.268612+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:56:31.011637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.133342+0000","last_change":"2024-03-25T02:00:33.707296+0000","last_active":"2024-03-25T02:35:04.133342+0000","last_peered":"2024-03-25T02:35:04.133342+0000","last_clean":"2024-03-25T02:35:04.133342+0000","last_became_active":"2024-03-25T02:00:33.691900+0000","last_became_peered":"2024-03-25T02:00:33.691900+0000","last_unstale":"2024-03-25T02:35:04.133342+0000","last_undegraded":"2024-03-25T02:35:04.133342+0000","last_fullsized":"2024-03-25T02:35:04.133342+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2412,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.286541+0000","last_change":"2024-03-25T02:01:39.897197+0000","last_active":"2024-03-25T02:35:04.286541+0000","last_peered":"2024-03-25T02:35:04.286541+0000","last_clean":"2024-03-25T02:35:04.286541+0000","last_became_active":"2024-03-25T02:01:39.890974+0000","last_became_peered":"2024-03-25T02:01:39.890974+0000","last_unstale":"2024-03-25T02:35:04.286541+0000","last_undegraded":"2024-03-25T02:35:04.286541+0000","last_fullsized":"2024-03-25T02:35:04.286541+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:25:57.375716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2419,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.278786+0000","last_change":"2024-03-25T02:00:33.702006+0000","last_active":"2024-03-25T02:35:04.278786+0000","last_peered":"2024-03-25T02:35:04.278786+0000","last_clean":"2024-03-25T02:35:04.278786+0000","last_became_active":"2024-03-25T02:00:33.693877+0000","last_became_peered":"2024-03-25T02:00:33.693877+0000","last_unstale":"2024-03-25T02:35:04.278786+0000","last_undegraded":"2024-03-25T02:35:04.278786+0000","last_fullsized":"2024-03-25T02:35:04.278786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:52:31.586659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.295044+0000","last_change":"2024-03-25T02:01:50.169634+0000","last_active":"2024-03-25T02:35:04.295044+0000","last_peered":"2024-03-25T02:35:04.295044+0000","last_clean":"2024-03-25T02:35:04.295044+0000","last_became_active":"2024-03-25T02:01:50.160922+0000","last_became_peered":"2024-03-25T02:01:50.160922+0000","last_unstale":"2024-03-25T02:35:04.295044+0000","last_undegraded":"2024-03-25T02:35:04.295044+0000","last_fullsized":"2024-03-25T02:35:04.295044+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:28:52.038897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.214999+0000","last_change":"2024-03-25T02:00:33.514196+0000","last_active":"2024-03-25T02:35:04.214999+0000","last_peered":"2024-03-25T02:35:04.214999+0000","last_clean":"2024-03-25T02:35:04.214999+0000","last_became_active":"2024-03-25T02:00:33.508976+0000","last_became_peered":"2024-03-25T02:00:33.508976+0000","last_unstale":"2024-03-25T02:35:04.214999+0000","last_undegraded":"2024-03-25T02:35:04.214999+0000","last_fullsized":"2024-03-25T02:35:04.214999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2416,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.282048+0000","last_change":"2024-03-25T02:01:59.814614+0000","last_active":"2024-03-25T02:35:04.282048+0000","last_peered":"2024-03-25T02:35:04.282048+0000","last_clean":"2024-03-25T02:35:04.282048+0000","last_became_active":"2024-03-25T02:01:59.807793+0000","last_became_peered":"2024-03-25T02:01:59.807793+0000","last_unstale":"2024-03-25T02:35:04.282048+0000","last_undegraded":"2024-03-25T02:35:04.282048+0000","last_fullsized":"2024-03-25T02:35:04.282048+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:42:56.823793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2425,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.006755+0000","last_change":"2024-03-25T02:00:33.937499+0000","last_active":"2024-03-25T02:35:04.006755+0000","last_peered":"2024-03-25T02:35:04.006755+0000","last_clean":"2024-03-25T02:35:04.006755+0000","last_became_active":"2024-03-25T02:00:33.909846+0000","last_became_peered":"2024-03-25T02:00:33.909846+0000","last_unstale":"2024-03-25T02:35:04.006755+0000","last_undegraded":"2024-03-25T02:35:04.006755+0000","last_fullsized":"2024-03-25T02:35:04.006755+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:26:26.798933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2408,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.254000+0000","last_change":"2024-03-25T02:01:37.864558+0000","last_active":"2024-03-25T02:35:04.254000+0000","last_peered":"2024-03-25T02:35:04.254000+0000","last_clean":"2024-03-25T02:35:04.254000+0000","last_became_active":"2024-03-25T02:01:37.856709+0000","last_became_peered":"2024-03-25T02:01:37.856709+0000","last_unstale":"2024-03-25T02:35:04.254000+0000","last_undegraded":"2024-03-25T02:35:04.254000+0000","last_fullsized":"2024-03-25T02:35:04.254000+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2416,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:12.153793+0000","last_change":"2024-03-25T02:00:33.801095+0000","last_active":"2024-03-25T02:35:12.153793+0000","last_peered":"2024-03-25T02:35:12.153793+0000","last_clean":"2024-03-25T02:35:12.153793+0000","last_became_active":"2024-03-25T02:00:33.792626+0000","last_became_peered":"2024-03-25T02:00:33.792626+0000","last_unstale":"2024-03-25T02:35:12.153793+0000","last_undegraded":"2024-03-25T02:35:12.153793+0000","last_fullsized":"2024-03-25T02:35:12.153793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:00:31.897474+0000","last_clean_scrub_stamp":"2024-03-25T02:00:31.897474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2411,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:03.946376+0000","last_change":"2024-03-25T02:01:54.523614+0000","last_active":"2024-03-25T02:35:03.946376+0000","last_peered":"2024-03-25T02:35:03.946376+0000","last_clean":"2024-03-25T02:35:03.946376+0000","last_became_active":"2024-03-25T02:01:54.518327+0000","last_became_peered":"2024-03-25T02:01:54.518327+0000","last_unstale":"2024-03-25T02:35:03.946376+0000","last_undegraded":"2024-03-25T02:35:03.946376+0000","last_fullsized":"2024-03-25T02:35:03.946376+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:28:31.375645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2407,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.328600+0000","last_change":"2024-03-25T02:01:53.995287+0000","last_active":"2024-03-25T02:35:04.328600+0000","last_peered":"2024-03-25T02:35:04.328600+0000","last_clean":"2024-03-25T02:35:04.328600+0000","last_became_active":"2024-03-25T02:01:53.989515+0000","last_became_peered":"2024-03-25T02:01:53.989515+0000","last_unstale":"2024-03-25T02:35:04.328600+0000","last_undegraded":"2024-03-25T02:35:04.328600+0000","last_fullsized":"2024-03-25T02:35:04.328600+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:43:06.616302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2408,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.356157+0000","last_change":"2024-03-25T02:01:55.429116+0000","last_active":"2024-03-25T02:35:04.356157+0000","last_peered":"2024-03-25T02:35:04.356157+0000","last_clean":"2024-03-25T02:35:04.356157+0000","last_became_active":"2024-03-25T02:01:55.419583+0000","last_became_peered":"2024-03-25T02:01:55.419583+0000","last_unstale":"2024-03-25T02:35:04.356157+0000","last_undegraded":"2024-03-25T02:35:04.356157+0000","last_fullsized":"2024-03-25T02:35:04.356157+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:54:42.345480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2409,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.277033+0000","last_change":"2024-03-25T02:01:56.827003+0000","last_active":"2024-03-25T02:35:04.277033+0000","last_peered":"2024-03-25T02:35:04.277033+0000","last_clean":"2024-03-25T02:35:04.277033+0000","last_became_active":"2024-03-25T02:01:56.799197+0000","last_became_peered":"2024-03-25T02:01:56.799197+0000","last_unstale":"2024-03-25T02:35:04.277033+0000","last_undegraded":"2024-03-25T02:35:04.277033+0000","last_fullsized":"2024-03-25T02:35:04.277033+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:49:03.924799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2405,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.349794+0000","last_change":"2024-03-25T02:01:50.245600+0000","last_active":"2024-03-25T02:35:04.349794+0000","last_peered":"2024-03-25T02:35:04.349794+0000","last_clean":"2024-03-25T02:35:04.349794+0000","last_became_active":"2024-03-25T02:01:50.226409+0000","last_became_peered":"2024-03-25T02:01:50.226409+0000","last_unstale":"2024-03-25T02:35:04.349794+0000","last_undegraded":"2024-03-25T02:35:04.349794+0000","last_fullsized":"2024-03-25T02:35:04.349794+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:33:00.764629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2408,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.265632+0000","last_change":"2024-03-25T02:01:39.447936+0000","last_active":"2024-03-25T02:35:04.265632+0000","last_peered":"2024-03-25T02:35:04.265632+0000","last_clean":"2024-03-25T02:35:04.265632+0000","last_became_active":"2024-03-25T02:01:39.437348+0000","last_became_peered":"2024-03-25T02:01:39.437348+0000","last_unstale":"2024-03-25T02:35:04.265632+0000","last_undegraded":"2024-03-25T02:35:04.265632+0000","last_fullsized":"2024-03-25T02:35:04.265632+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:31:57.241440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.278919+0000","last_change":"2024-03-25T02:01:43.296920+0000","last_active":"2024-03-25T02:35:04.278919+0000","last_peered":"2024-03-25T02:35:04.278919+0000","last_clean":"2024-03-25T02:35:04.278919+0000","last_became_active":"2024-03-25T02:01:43.285633+0000","last_became_peered":"2024-03-25T02:01:43.285633+0000","last_unstale":"2024-03-25T02:35:04.278919+0000","last_undegraded":"2024-03-25T02:35:04.278919+0000","last_fullsized":"2024-03-25T02:35:04.278919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:17:06.380180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2408,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.257683+0000","last_change":"2024-03-25T02:01:41.663556+0000","last_active":"2024-03-25T02:35:04.257683+0000","last_peered":"2024-03-25T02:35:04.257683+0000","last_clean":"2024-03-25T02:35:04.257683+0000","last_became_active":"2024-03-25T02:01:41.631996+0000","last_became_peered":"2024-03-25T02:01:41.631996+0000","last_unstale":"2024-03-25T02:35:04.257683+0000","last_undegraded":"2024-03-25T02:35:04.257683+0000","last_fullsized":"2024-03-25T02:35:04.257683+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:56:07.666478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2408,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.167583+0000","last_change":"2024-03-25T02:01:44.130872+0000","last_active":"2024-03-25T02:35:04.167583+0000","last_peered":"2024-03-25T02:35:04.167583+0000","last_clean":"2024-03-25T02:35:04.167583+0000","last_became_active":"2024-03-25T02:01:44.123357+0000","last_became_peered":"2024-03-25T02:01:44.123357+0000","last_unstale":"2024-03-25T02:35:04.167583+0000","last_undegraded":"2024-03-25T02:35:04.167583+0000","last_fullsized":"2024-03-25T02:35:04.167583+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2451,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:10.848157+0000","last_change":"2024-03-25T02:35:10.848953+0000","last_active":"2024-03-25T02:35:10.848157+0000","last_peered":"2024-03-25T02:35:10.848157+0000","last_clean":"2024-03-25T02:35:10.848157+0000","last_became_active":"2024-03-25T02:01:40.275307+0000","last_became_peered":"2024-03-25T02:01:40.275307+0000","last_unstale":"2024-03-25T02:35:10.848157+0000","last_undegraded":"2024-03-25T02:35:10.848157+0000","last_fullsized":"2024-03-25T02:35:10.848157+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:10.846551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:10.846551+0000","last_clean_scrub_stamp":"2024-03-25T02:35:10.846551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:22:28.639302+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2416,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.291957+0000","last_change":"2024-03-25T02:01:50.267025+0000","last_active":"2024-03-25T02:35:04.291957+0000","last_peered":"2024-03-25T02:35:04.291957+0000","last_clean":"2024-03-25T02:35:04.291957+0000","last_became_active":"2024-03-25T02:01:50.240201+0000","last_became_peered":"2024-03-25T02:01:50.240201+0000","last_unstale":"2024-03-25T02:35:04.291957+0000","last_undegraded":"2024-03-25T02:35:04.291957+0000","last_fullsized":"2024-03-25T02:35:04.291957+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:09:56.028592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2410,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.275273+0000","last_change":"2024-03-25T02:01:56.819977+0000","last_active":"2024-03-25T02:35:04.275273+0000","last_peered":"2024-03-25T02:35:04.275273+0000","last_clean":"2024-03-25T02:35:04.275273+0000","last_became_active":"2024-03-25T02:01:56.787332+0000","last_became_peered":"2024-03-25T02:01:56.787332+0000","last_unstale":"2024-03-25T02:35:04.275273+0000","last_undegraded":"2024-03-25T02:35:04.275273+0000","last_fullsized":"2024-03-25T02:35:04.275273+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:10:10.439975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2407,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:04.200157+0000","last_change":"2024-03-25T02:01:40.340111+0000","last_active":"2024-03-25T02:35:04.200157+0000","last_peered":"2024-03-25T02:35:04.200157+0000","last_clean":"2024-03-25T02:35:04.200157+0000","last_became_active":"2024-03-25T02:01:40.334290+0000","last_became_peered":"2024-03-25T02:01:40.334290+0000","last_unstale":"2024-03-25T02:35:04.200157+0000","last_undegraded":"2024-03-25T02:35:04.200157+0000","last_fullsized":"2024-03-25T02:35:04.200157+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:01:34.413541+0000","last_clean_scrub_stamp":"2024-03-25T02:01:34.413541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":20,"seq":85899346336,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70932,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300908,"statfs":{"total":96636764160,"available":96564129792,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346342,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71320,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300520,"statfs":{"total":96636764160,"available":96563732480,"internally_reserved":0,"allocated":4141056,"data_stored":1936955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379044,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70936,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300904,"statfs":{"total":96636764160,"available":96564125696,"internally_reserved":0,"allocated":3743744,"data_stored":1539120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379046,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66768,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305072,"statfs":{"total":96636764160,"available":96568393728,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71316,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300524,"statfs":{"total":96636764160,"available":96563736576,"internally_reserved":0,"allocated":4141056,"data_stored":1936955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509859,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66768,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305072,"statfs":{"total":96636764160,"available":96568393728,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70924,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300916,"statfs":{"total":96636764160,"available":96564137984,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542566,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70936,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300904,"statfs":{"total":96636764160,"available":96564125696,"internally_reserved":0,"allocated":3743744,"data_stored":1539120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-25T02:35:16.073 INFO:tasks.ceph:pgid 9.19 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.073 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.073 INFO:tasks.ceph:pgid 9.1b last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.1a last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.074 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-25T02:00:15.236797+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=15, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-25T02:00:31.897474+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=31, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 9.8 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 9.b last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-03-25T02:00:31.897474+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=31, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 9.a last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-25T02:00:31.897474+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=31, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 9.9 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-03-25T02:00:31.897474+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=31, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.075 INFO:tasks.ceph:pgid 9.f last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-25T02:00:31.897474+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=31, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 9.e last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-25T02:00:31.897474+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=31, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 9.d last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-25T02:00:31.897474+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=0, tm_sec=31, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 9.1e last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 9.1f last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.076 INFO:tasks.ceph:pgid 9.1c last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.1d last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.12 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.11 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.17 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2024-03-25T02:01:34.413541+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=1, tm_sec=34, tm_wday=0, tm_yday=85, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=25, tm_hour=2, tm_min=35, tm_sec=7, tm_wday=0, tm_yday=85, tm_isdst=0) 2024-03-25T02:35:16.077 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-25T02:35:23.883 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:06.732 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.886 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:06.736 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.905 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:06.754 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.934 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:06.739 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.935 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:06.784 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.935 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:06.741 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.936 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:06.785 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.951 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:06.756 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:23.954 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:06.803 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.895 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:11.744 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.899 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:11.748 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.912 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:11.761 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.947 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:11.752 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.948 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:11.753 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.949 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:11.798 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.950 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:11.799 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.958 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:11.763 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:28.960 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:11.809 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:36.078 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-25T02:35:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.191+0000 7fd2b2c4e640 10 monclient: get_monmap_and_config 2024-03-25T02:35:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.191+0000 7fd2b2c4e640 10 monclient: build_initial_monmap 2024-03-25T02:35:36.180 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.191+0000 7fd2b2c4e640 10 monclient: monmap: 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:36.192938+0000 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:36.192938+0000 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:36.181 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 1 Processor -- start 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 1 -- start start 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient: init 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd2ac0af870 0x7fd2ac13e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ac0af870 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2ac13e850 addr v1:172.21.15.83:6789/0 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0b0a60 0x7fd2ac141740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.184 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac0b0a60 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 1 -- --> v1:172.21.15.83:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ac0b0610 con 0x7fd2ac13e850 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fd2ac0ae820 con 0x7fd2ac0af870 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ac141c80 con 0x7fd2ac0b0a60 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.192+0000 7fd2b2c4e640 10 monclient(hunting): authenticate will time out at 3455.382851s 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2b11eb640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0b0a60 0x7fd2ac141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2abfff640 1 --1- >> v1:172.21.15.83:6789/0 conn(0x7fd2ac13e850 0x7fd2ac141030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.83:6789/0 says I am v1:172.21.15.83:56894/0 (socket says 172.21.15.83:56894) 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2abfff640 1 -- 172.21.15.83:0/2056319779 learned_addr learned my addr 172.21.15.83:0/2056319779 (peer_addr_for_me v1:172.21.15.83:0/0) 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2b09ea640 1 --2- 172.21.15.83:0/2056319779 >> v2:172.21.15.153:3300/0 conn(0x7fd2ac0af870 0x7fd2ac13e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.185 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2b11eb640 10 monclient(hunting): get_auth_request con 0x7fd2ac0b0a60 auth_method 0 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2b11eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2b11eb640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.193+0000 7fd2b11eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b09ea640 10 monclient(hunting): get_auth_request con 0x7fd2ac0af870 auth_method 0 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b09ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b09ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b09ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b11eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b11eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b11eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b09ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.186 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b09ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.194+0000 7fd2b09ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.187 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.195+0000 7fd2ab7fe640 1 -- 172.21.15.83:0/2056319779 <== mon.0 v1:172.21.15.83:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1297915261 0 0) 0x7fd2ac0b0610 con 0x7fd2ac13e850 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.195+0000 7fd2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.195+0000 7fd2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.195+0000 7fd2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.195+0000 7fd2ab7fe640 10 monclient(hunting): my global_id is 15468 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2ab7fe640 1 -- 172.21.15.83:0/2056319779 --> v1:172.21.15.83:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd29c003520 con 0x7fd2ac13e850 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 10 monclient(hunting): handle_auth_done global_id 17297 payload 274 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 1 -- 172.21.15.83:0/2056319779 >> v2:172.21.15.153:3300/0 conn(0x7fd2ac0af870 msgr2=0x7fd2ac13e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 1 --2- 172.21.15.83:0/2056319779 >> v2:172.21.15.153:3300/0 conn(0x7fd2ac0af870 0x7fd2ac13e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 1 -- 172.21.15.83:0/2056319779 >> v1:172.21.15.83:6789/0 conn(0x7fd2ac13e850 legacy=0x7fd2ac141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 10 monclient: _finish_hunting 0 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 1 monclient: found mon.noname-b 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:36.188 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.196+0000 7fd2b11eb640 1 -- 172.21.15.83:0/2056319779 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ac0afc50 con 0x7fd2ac0b0a60 2024-03-25T02:35:36.189 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.197+0000 7fd2b11eb640 1 --2- 172.21.15.83:0/2056319779 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0b0a60 0x7fd2ac141740 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95d09a7d0d5cc2c server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 1 -- 172.21.15.83:0/2056319779 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4039070 con 0x7fd2ac0b0a60 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: dump: 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.190 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: _finish_auth 0 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 1 -- 172.21.15.83:0/2056319779 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a402cbc0 con 0x7fd2ac0b0a60 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2b2c4e640 5 monclient: authenticate success, global_id 17297 2024-03-25T02:35:36.191 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2b2c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 1 -- 172.21.15.83:0/2056319779 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4034440 con 0x7fd2ac0b0a60 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2ab7fe640 10 monclient: dump: 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.192 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.193 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2b2c4e640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:36.193 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.201+0000 7fd2b2c4e640 4 set_mon_vals no callback set 2024-03-25T02:35:36.196 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 10 monclient: shutdown 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/2056319779 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0b0a60 msgr2=0x7fd2ac141740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/2056319779 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0b0a60 0x7fd2ac141740 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a402be10 comp rx=0 tx=0).stop 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/2056319779 shutdown_connections 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/2056319779 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0b0a60 0x7fd2ac141740 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/2056319779 >> v2:172.21.15.153:3300/0 conn(0x7fd2ac0af870 0x7fd2ac13e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/2056319779 >> 172.21.15.83:0/2056319779 conn(0x7fd2ac0b4a10 msgr2=0x7fd2ac0b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/2056319779 shutdown_connections 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/2056319779 wait complete. 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 10 monclient: build_initial_monmap 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.202+0000 7fd2b2c4e640 10 monclient: monmap: 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:36.197 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.203+0000 7fd2b2c4e640 1 librados: starting msgr at 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.203+0000 7fd2b2c4e640 1 librados: starting objecter 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.203+0000 7fd2b2c4e640 1 Processor -- start 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.203+0000 7fd2b2c4e640 1 -- start start 2024-03-25T02:35:36.198 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.203+0000 7fd2b2c4e640 1 librados: setting wanted keys 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.203+0000 7fd2b2c4e640 1 librados: calling monclient init 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.203+0000 7fd2b2c4e640 10 monclient: init 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0af870 0x7fd2ac1424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ac0af870 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0b0a60 0x7fd2ac144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac0b0a60 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2ac13e850 0x7fd2ac144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd2ac13e850 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac14bb60 con 0x7fd2ac13e850 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.199 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac14b9e0 con 0x7fd2ac0b0a60 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ac14bce0 con 0x7fd2ac0af870 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b2c4e640 10 monclient(hunting): authenticate will time out at 3455.394078s 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b09ea640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0af870 0x7fd2ac1424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2abfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0b0a60 0x7fd2ac144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b11eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2ac13e850 0x7fd2ac144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b09ea640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0af870 0x7fd2ac1424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:51858/0 (socket says 172.21.15.83:51858) 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b09ea640 1 -- 172.21.15.83:0/3893891843 learned_addr learned my addr 172.21.15.83:0/3893891843 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b09ea640 10 monclient(hunting): get_auth_request con 0x7fd2ac0af870 auth_method 0 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b09ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b09ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.204+0000 7fd2b09ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b11eb640 10 monclient(hunting): get_auth_request con 0x7fd2ac13e850 auth_method 0 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b11eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b11eb640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b11eb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2abfff640 10 monclient(hunting): get_auth_request con 0x7fd2ac0b0a60 auth_method 0 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b09ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b09ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b09ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.201 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b11eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b11eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.205+0000 7fd2b11eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 10 monclient(hunting): handle_auth_done global_id 17300 payload 824 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 1 -- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0b0a60 msgr2=0x7fd2ac144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0b0a60 0x7fd2ac144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 1 -- 172.21.15.83:0/3893891843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2ac13e850 msgr2=0x7fd2ac144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2ac13e850 0x7fd2ac144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 10 monclient: _finish_hunting 0 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 1 monclient: found mon.noname-c 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.202 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.208+0000 7fd2b09ea640 1 -- 172.21.15.83:0/3893891843 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ac145180 con 0x7fd2ac0af870 2024-03-25T02:35:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.210+0000 7fd2b09ea640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0af870 0x7fd2ac1424b0 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000e9e0 tx=0x7fd2a000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.210+0000 7fd2a97fa640 10 client.?.objecter ms_handle_connect 0x7fd2ac0af870 2024-03-25T02:35:36.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.210+0000 7fd2a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2a97fa640 1 -- 172.21.15.83:0/3893891843 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0017070 con 0x7fd2ac0af870 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2a97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2a97fa640 10 monclient: dump: 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.205 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2a97fa640 10 monclient: _finish_auth 0 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2a97fa640 10 monclient: _check_auth_tickets 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 5 monclient: authenticate success, global_id 17300 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 10 monclient: _renew_subs 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ac143f90 con 0x7fd2ac0af870 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 10 client.17300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 10 monclient: _renew_subs 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ac1429f0 con 0x7fd2ac0af870 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.212+0000 7fd2b2c4e640 1 librados: init done 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2b2c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:36.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2b2c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2b2c4e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2ac075810 con 0x7fd2ac0af870 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2a97fa640 1 -- 172.21.15.83:0/3893891843 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a0005990 con 0x7fd2ac0af870 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2a97fa640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2a97fa640 1 -- 172.21.15.83:0/3893891843 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0021060 con 0x7fd2ac0af870 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2a97fa640 10 monclient: dump: 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.207 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.208 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.208 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.213+0000 7fd2aaffd640 4 set_mon_vals no callback set 2024-03-25T02:35:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.457+0000 7fd2a97fa640 1 -- 172.21.15.83:0/3893891843 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fd2a001a070 con 0x7fd2ac0af870 2024-03-25T02:35:36.446 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.457+0000 7fd2a97fa640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd29006da20 0x7fd29006fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.458+0000 7fd2abfff640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd29006da20 0x7fd29006fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.458+0000 7fd2abfff640 10 monclient: get_auth_request con 0x7fd29006da20 auth_method 0 2024-03-25T02:35:36.447 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.459+0000 7fd2a97fa640 10 client.17300.objecter ms_handle_connect 0x7fd29006da20 2024-03-25T02:35:36.448 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.459+0000 7fd2abfff640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd29006da20 0x7fd29006fee0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd298005880 tx=0x7fd298009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 1 -- 172.21.15.83:0/3893891843 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7fd2a0057b20 con 0x7fd2ac0af870 2024-03-25T02:35:36.460 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 10 client.17300.objecter ms_dispatch 0x7fd2ac000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 3 client.17300.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 3 client.17300.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 20 client.17300.objecter dump_active .. 0 homeless 2024-03-25T02:35:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 1 -- 172.21.15.83:0/3893891843 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd2a0059a80 con 0x7fd2ac0af870 2024-03-25T02:35:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:36.461 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.471+0000 7fd2a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:36.641 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.652+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 --> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2ac076d20 con 0x7fd29006da20 2024-03-25T02:35:36.644 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.655+0000 7fd2a97fa640 1 -- 172.21.15.83:0/3893891843 <== mgr.4111 v2:172.21.15.153:6800/4072599690 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116158 (secure 0 0 0) 0x7fd29800f820 con 0x7fd29006da20 2024-03-25T02:35:36.646 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-25T02:35:36.646 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-03-25T02:35:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 10 librados: watch_flush enter 2024-03-25T02:35:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 10 librados: watch_flush exit 2024-03-25T02:35:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 20 client.17300.objecter shutdown clearing up homeless session... 2024-03-25T02:35:36.648 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 10 client.17300.objecter successfully canceled tick 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd29006da20 msgr2=0x7fd29006fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd29006da20 0x7fd29006fee0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd298005880 tx=0x7fd298009040 comp rx=0 tx=0).stop 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 10 monclient: shutdown 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0af870 msgr2=0x7fd2ac1424b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.657+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0af870 0x7fd2ac1424b0 secure :-1 s=READY pgs=3210 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000e9e0 tx=0x7fd2a000eea0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 shutdown_connections 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fd29006da20 0x7fd29006fee0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd2ac13e850 0x7fd2ac144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0b0a60 0x7fd2ac144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 --2- 172.21.15.83:0/3893891843 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fd2ac0af870 0x7fd2ac1424b0 unknown :-1 s=CLOSED pgs=3210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 >> 172.21.15.83:0/3893891843 conn(0x7fd2ac0b4a10 msgr2=0x7fd2ac0b4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 shutdown_connections 2024-03-25T02:35:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 -- 172.21.15.83:0/3893891843 wait complete. 2024-03-25T02:35:36.650 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.658+0000 7fd2b2c4e640 1 librados: shutdown 2024-03-25T02:35:36.657 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":2314,"stamp":"2024-03-25T02:35:34.938051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":89,"num_write":121,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":559864,"kb_used_data":30008,"kb_used_omap":12,"kb_used_meta":529779,"kb_avail":754414856,"statfs":{"total":773094113280,"available":772520812544,"internally_reserved":0,"allocated":30728192,"data_stored":13108610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":542494256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003150"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":2414,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:19.250025+0000","last_change":"2024-03-25T02:35:19.250872+0000","last_active":"2024-03-25T02:35:19.250025+0000","last_peered":"2024-03-25T02:35:19.250025+0000","last_clean":"2024-03-25T02:35:19.250025+0000","last_became_active":"2024-03-25T02:01:59.217800+0000","last_became_peered":"2024-03-25T02:01:59.217800+0000","last_unstale":"2024-03-25T02:35:19.250025+0000","last_undegraded":"2024-03-25T02:35:19.250025+0000","last_fullsized":"2024-03-25T02:35:19.250025+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:19.248403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:19.248403+0000","last_clean_scrub_stamp":"2024-03-25T02:35:19.248403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:48:25.126343+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":2425,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:13.681681+0000","last_change":"2024-03-25T02:35:13.682430+0000","last_active":"2024-03-25T02:35:13.681681+0000","last_peered":"2024-03-25T02:35:13.681681+0000","last_clean":"2024-03-25T02:35:13.681681+0000","last_became_active":"2024-03-25T02:01:45.737540+0000","last_became_peered":"2024-03-25T02:01:45.737540+0000","last_unstale":"2024-03-25T02:35:13.681681+0000","last_undegraded":"2024-03-25T02:35:13.681681+0000","last_fullsized":"2024-03-25T02:35:13.681681+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:13.680175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:13.680175+0000","last_clean_scrub_stamp":"2024-03-25T02:35:13.680175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:10:00.964512+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":2422,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:14.754198+0000","last_change":"2024-03-25T02:35:14.754962+0000","last_active":"2024-03-25T02:35:14.754198+0000","last_peered":"2024-03-25T02:35:14.754198+0000","last_clean":"2024-03-25T02:35:14.754198+0000","last_became_active":"2024-03-25T02:01:57.014842+0000","last_became_peered":"2024-03-25T02:01:57.014842+0000","last_unstale":"2024-03-25T02:35:14.754198+0000","last_undegraded":"2024-03-25T02:35:14.754198+0000","last_fullsized":"2024-03-25T02:35:14.754198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:14.752563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:14.752563+0000","last_clean_scrub_stamp":"2024-03-25T02:35:14.752563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:19:42.498028+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:16.260023+0000","last_change":"2024-03-25T02:35:16.260802+0000","last_active":"2024-03-25T02:35:16.260023+0000","last_peered":"2024-03-25T02:35:16.260023+0000","last_clean":"2024-03-25T02:35:16.260023+0000","last_became_active":"2024-03-25T02:01:39.486234+0000","last_became_peered":"2024-03-25T02:01:39.486234+0000","last_unstale":"2024-03-25T02:35:16.260023+0000","last_undegraded":"2024-03-25T02:35:16.260023+0000","last_fullsized":"2024-03-25T02:35:16.260023+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:16.258371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:16.258371+0000","last_clean_scrub_stamp":"2024-03-25T02:35:16.258371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:58:11.543004+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":2423,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:17.284550+0000","last_change":"2024-03-25T02:35:17.285362+0000","last_active":"2024-03-25T02:35:17.284550+0000","last_peered":"2024-03-25T02:35:17.284550+0000","last_clean":"2024-03-25T02:35:17.284550+0000","last_became_active":"2024-03-25T02:01:39.792053+0000","last_became_peered":"2024-03-25T02:01:39.792053+0000","last_unstale":"2024-03-25T02:35:17.284550+0000","last_undegraded":"2024-03-25T02:35:17.284550+0000","last_fullsized":"2024-03-25T02:35:17.284550+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:17.282931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:17.282931+0000","last_clean_scrub_stamp":"2024-03-25T02:35:17.282931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:40:55.440062+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":2422,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:17.537764+0000","last_change":"2024-03-25T02:35:17.538556+0000","last_active":"2024-03-25T02:35:17.537764+0000","last_peered":"2024-03-25T02:35:17.537764+0000","last_clean":"2024-03-25T02:35:17.537764+0000","last_became_active":"2024-03-25T02:01:42.877188+0000","last_became_peered":"2024-03-25T02:01:42.877188+0000","last_unstale":"2024-03-25T02:35:17.537764+0000","last_undegraded":"2024-03-25T02:35:17.537764+0000","last_fullsized":"2024-03-25T02:35:17.537764+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:17.536138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:17.536138+0000","last_clean_scrub_stamp":"2024-03-25T02:35:17.536138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T13:52:33.567887+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":2427,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:12.721647+0000","last_change":"2024-03-25T02:35:12.722430+0000","last_active":"2024-03-25T02:35:12.721647+0000","last_peered":"2024-03-25T02:35:12.721647+0000","last_clean":"2024-03-25T02:35:12.721647+0000","last_became_active":"2024-03-25T02:01:45.785913+0000","last_became_peered":"2024-03-25T02:01:45.785913+0000","last_unstale":"2024-03-25T02:35:12.721647+0000","last_undegraded":"2024-03-25T02:35:12.721647+0000","last_fullsized":"2024-03-25T02:35:12.721647+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:12.720046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:12.720046+0000","last_clean_scrub_stamp":"2024-03-25T02:35:12.720046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:56.718776+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":2426,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:14.760832+0000","last_change":"2024-03-25T02:35:14.761635+0000","last_active":"2024-03-25T02:35:14.760832+0000","last_peered":"2024-03-25T02:35:14.760832+0000","last_clean":"2024-03-25T02:35:14.760832+0000","last_became_active":"2024-03-25T02:01:47.939684+0000","last_became_peered":"2024-03-25T02:01:47.939684+0000","last_unstale":"2024-03-25T02:35:14.760832+0000","last_undegraded":"2024-03-25T02:35:14.760832+0000","last_fullsized":"2024-03-25T02:35:14.760832+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:14.759190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:14.759190+0000","last_clean_scrub_stamp":"2024-03-25T02:35:14.759190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:49.598140+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"1250'8","reported_seq":2437,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:19.605480+0000","last_change":"2024-03-25T02:35:19.606677+0000","last_active":"2024-03-25T02:35:19.605480+0000","last_peered":"2024-03-25T02:35:19.605480+0000","last_clean":"2024-03-25T02:35:19.605480+0000","last_became_active":"2024-03-25T02:01:58.306972+0000","last_became_peered":"2024-03-25T02:01:58.306972+0000","last_unstale":"2024-03-25T02:35:19.605480+0000","last_undegraded":"2024-03-25T02:35:19.605480+0000","last_fullsized":"2024-03-25T02:35:19.605480+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"1250'8","last_scrub_stamp":"2024-03-25T02:35:19.603879+0000","last_deep_scrub":"1250'8","last_deep_scrub_stamp":"2024-03-25T02:35:19.603879+0000","last_clean_scrub_stamp":"2024-03-25T02:35:19.603879+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:24:27.125372+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":2423,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:20.314757+0000","last_change":"2024-03-25T02:35:20.315565+0000","last_active":"2024-03-25T02:35:20.314757+0000","last_peered":"2024-03-25T02:35:20.314757+0000","last_clean":"2024-03-25T02:35:20.314757+0000","last_became_active":"2024-03-25T02:01:44.731330+0000","last_became_peered":"2024-03-25T02:01:44.731330+0000","last_unstale":"2024-03-25T02:35:20.314757+0000","last_undegraded":"2024-03-25T02:35:20.314757+0000","last_fullsized":"2024-03-25T02:35:20.314757+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:20.313137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:20.313137+0000","last_clean_scrub_stamp":"2024-03-25T02:35:20.313137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T14:10:50.764217+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":2455,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:09.893335+0000","last_change":"2024-03-25T02:35:09.893335+0000","last_active":"2024-03-25T02:35:09.893335+0000","last_peered":"2024-03-25T02:35:09.893335+0000","last_clean":"2024-03-25T02:35:09.893335+0000","last_became_active":"2024-03-25T02:01:38.008862+0000","last_became_peered":"2024-03-25T02:01:38.008862+0000","last_unstale":"2024-03-25T02:35:09.893335+0000","last_undegraded":"2024-03-25T02:35:09.893335+0000","last_fullsized":"2024-03-25T02:35:09.893335+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:09.890709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:09.890709+0000","last_clean_scrub_stamp":"2024-03-25T02:35:09.890709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:39:18.872661+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:18.215373+0000","last_change":"2024-03-25T02:35:18.216174+0000","last_active":"2024-03-25T02:35:18.215373+0000","last_peered":"2024-03-25T02:35:18.215373+0000","last_clean":"2024-03-25T02:35:18.215373+0000","last_became_active":"2024-03-25T02:01:55.452605+0000","last_became_peered":"2024-03-25T02:01:55.452605+0000","last_unstale":"2024-03-25T02:35:18.215373+0000","last_undegraded":"2024-03-25T02:35:18.215373+0000","last_fullsized":"2024-03-25T02:35:18.215373+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:18.213718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:18.213718+0000","last_clean_scrub_stamp":"2024-03-25T02:35:18.213718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:55:15.677400+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"1250'79","reported_seq":2525,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:11.460256+0000","last_change":"2024-03-25T02:35:11.461845+0000","last_active":"2024-03-25T02:35:11.460256+0000","last_peered":"2024-03-25T02:35:11.460256+0000","last_clean":"2024-03-25T02:35:11.460256+0000","last_became_active":"2024-03-25T02:00:25.840109+0000","last_became_peered":"2024-03-25T02:00:25.840109+0000","last_unstale":"2024-03-25T02:35:11.460256+0000","last_undegraded":"2024-03-25T02:35:11.460256+0000","last_fullsized":"2024-03-25T02:35:11.460256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"1250'79","last_scrub_stamp":"2024-03-25T02:35:11.458032+0000","last_deep_scrub":"1250'79","last_deep_scrub_stamp":"2024-03-25T02:35:11.458032+0000","last_clean_scrub_stamp":"2024-03-25T02:35:11.458032+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:42:16.316849+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2426,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:16.563563+0000","last_change":"2024-03-25T02:35:16.564306+0000","last_active":"2024-03-25T02:35:16.563563+0000","last_peered":"2024-03-25T02:35:16.563563+0000","last_clean":"2024-03-25T02:35:16.563563+0000","last_became_active":"2024-03-25T02:00:33.865870+0000","last_became_peered":"2024-03-25T02:00:33.865870+0000","last_unstale":"2024-03-25T02:35:16.563563+0000","last_undegraded":"2024-03-25T02:35:16.563563+0000","last_fullsized":"2024-03-25T02:35:16.563563+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:16.562008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:16.562008+0000","last_clean_scrub_stamp":"2024-03-25T02:35:16.562008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:27:32.030605+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":2422,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:19.285339+0000","last_change":"2024-03-25T02:35:19.286157+0000","last_active":"2024-03-25T02:35:19.285339+0000","last_peered":"2024-03-25T02:35:19.285339+0000","last_clean":"2024-03-25T02:35:19.285339+0000","last_became_active":"2024-03-25T02:01:42.904032+0000","last_became_peered":"2024-03-25T02:01:42.904032+0000","last_unstale":"2024-03-25T02:35:19.285339+0000","last_undegraded":"2024-03-25T02:35:19.285339+0000","last_fullsized":"2024-03-25T02:35:19.285339+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:19.283762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:19.283762+0000","last_clean_scrub_stamp":"2024-03-25T02:35:19.283762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:25:53.389892+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2427,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:12.561114+0000","last_change":"2024-03-25T02:35:12.561880+0000","last_active":"2024-03-25T02:35:12.561114+0000","last_peered":"2024-03-25T02:35:12.561114+0000","last_clean":"2024-03-25T02:35:12.561114+0000","last_became_active":"2024-03-25T02:00:33.356691+0000","last_became_peered":"2024-03-25T02:00:33.356691+0000","last_unstale":"2024-03-25T02:35:12.561114+0000","last_undegraded":"2024-03-25T02:35:12.561114+0000","last_fullsized":"2024-03-25T02:35:12.561114+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:12.559543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:12.559543+0000","last_clean_scrub_stamp":"2024-03-25T02:35:12.559543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T14:13:30.640010+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":2422,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:15.332702+0000","last_change":"2024-03-25T02:35:15.333472+0000","last_active":"2024-03-25T02:35:15.332702+0000","last_peered":"2024-03-25T02:35:15.332702+0000","last_clean":"2024-03-25T02:35:15.332702+0000","last_became_active":"2024-03-25T02:01:39.122271+0000","last_became_peered":"2024-03-25T02:01:39.122271+0000","last_unstale":"2024-03-25T02:35:15.332702+0000","last_undegraded":"2024-03-25T02:35:15.332702+0000","last_fullsized":"2024-03-25T02:35:15.332702+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:15.331124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:15.331124+0000","last_clean_scrub_stamp":"2024-03-25T02:35:15.331124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:02:49.317480+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2431,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:16.493863+0000","last_change":"2024-03-25T02:35:16.494632+0000","last_active":"2024-03-25T02:35:16.493863+0000","last_peered":"2024-03-25T02:35:16.493863+0000","last_clean":"2024-03-25T02:35:16.493863+0000","last_became_active":"2024-03-25T02:00:33.429925+0000","last_became_peered":"2024-03-25T02:00:33.429925+0000","last_unstale":"2024-03-25T02:35:16.493863+0000","last_undegraded":"2024-03-25T02:35:16.493863+0000","last_fullsized":"2024-03-25T02:35:16.493863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:16.492245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:16.492245+0000","last_clean_scrub_stamp":"2024-03-25T02:35:16.492245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:09:23.716755+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:14.395358+0000","last_change":"2024-03-25T02:35:14.396149+0000","last_active":"2024-03-25T02:35:14.395358+0000","last_peered":"2024-03-25T02:35:14.395358+0000","last_clean":"2024-03-25T02:35:14.395358+0000","last_became_active":"2024-03-25T02:01:53.306467+0000","last_became_peered":"2024-03-25T02:01:53.306467+0000","last_unstale":"2024-03-25T02:35:14.395358+0000","last_undegraded":"2024-03-25T02:35:14.395358+0000","last_fullsized":"2024-03-25T02:35:14.395358+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:14.393811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:14.393811+0000","last_clean_scrub_stamp":"2024-03-25T02:35:14.393811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:15:05.394583+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2426,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:14.578913+0000","last_change":"2024-03-25T02:35:14.579715+0000","last_active":"2024-03-25T02:35:14.578913+0000","last_peered":"2024-03-25T02:35:14.578913+0000","last_clean":"2024-03-25T02:35:14.578913+0000","last_became_active":"2024-03-25T02:00:33.691900+0000","last_became_peered":"2024-03-25T02:00:33.691900+0000","last_unstale":"2024-03-25T02:35:14.578913+0000","last_undegraded":"2024-03-25T02:35:14.578913+0000","last_fullsized":"2024-03-25T02:35:14.578913+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:14.577364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:14.577364+0000","last_clean_scrub_stamp":"2024-03-25T02:35:14.577364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T10:23:47.700649+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":2421,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:18.309947+0000","last_change":"2024-03-25T02:35:18.310706+0000","last_active":"2024-03-25T02:35:18.309947+0000","last_peered":"2024-03-25T02:35:18.309947+0000","last_clean":"2024-03-25T02:35:18.309947+0000","last_became_active":"2024-03-25T02:01:39.890974+0000","last_became_peered":"2024-03-25T02:01:39.890974+0000","last_unstale":"2024-03-25T02:35:18.309947+0000","last_undegraded":"2024-03-25T02:35:18.309947+0000","last_fullsized":"2024-03-25T02:35:18.309947+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:18.308334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:18.308334+0000","last_clean_scrub_stamp":"2024-03-25T02:35:18.308334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:00:44.279365+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2428,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:14.342517+0000","last_change":"2024-03-25T02:35:14.343300+0000","last_active":"2024-03-25T02:35:14.342517+0000","last_peered":"2024-03-25T02:35:14.342517+0000","last_clean":"2024-03-25T02:35:14.342517+0000","last_became_active":"2024-03-25T02:00:33.693877+0000","last_became_peered":"2024-03-25T02:00:33.693877+0000","last_unstale":"2024-03-25T02:35:14.342517+0000","last_undegraded":"2024-03-25T02:35:14.342517+0000","last_fullsized":"2024-03-25T02:35:14.342517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:14.340979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:14.340979+0000","last_clean_scrub_stamp":"2024-03-25T02:35:14.340979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:34:11.245217+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":2426,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:15.808703+0000","last_change":"2024-03-25T02:35:15.809448+0000","last_active":"2024-03-25T02:35:15.808703+0000","last_peered":"2024-03-25T02:35:15.808703+0000","last_clean":"2024-03-25T02:35:15.808703+0000","last_became_active":"2024-03-25T02:01:50.160922+0000","last_became_peered":"2024-03-25T02:01:50.160922+0000","last_unstale":"2024-03-25T02:35:15.808703+0000","last_undegraded":"2024-03-25T02:35:15.808703+0000","last_fullsized":"2024-03-25T02:35:15.808703+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:15.807240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:15.807240+0000","last_clean_scrub_stamp":"2024-03-25T02:35:15.807240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:46:21.499909+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2426,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:13.603344+0000","last_change":"2024-03-25T02:35:13.604092+0000","last_active":"2024-03-25T02:35:13.603344+0000","last_peered":"2024-03-25T02:35:13.603344+0000","last_clean":"2024-03-25T02:35:13.603344+0000","last_became_active":"2024-03-25T02:00:33.508976+0000","last_became_peered":"2024-03-25T02:00:33.508976+0000","last_unstale":"2024-03-25T02:35:13.603344+0000","last_undegraded":"2024-03-25T02:35:13.603344+0000","last_fullsized":"2024-03-25T02:35:13.603344+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:13.601825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:13.601825+0000","last_clean_scrub_stamp":"2024-03-25T02:35:13.601825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:04:32.872410+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":2425,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:17.789144+0000","last_change":"2024-03-25T02:35:17.789934+0000","last_active":"2024-03-25T02:35:17.789144+0000","last_peered":"2024-03-25T02:35:17.789144+0000","last_clean":"2024-03-25T02:35:17.789144+0000","last_became_active":"2024-03-25T02:01:59.807793+0000","last_became_peered":"2024-03-25T02:01:59.807793+0000","last_unstale":"2024-03-25T02:35:17.789144+0000","last_undegraded":"2024-03-25T02:35:17.789144+0000","last_fullsized":"2024-03-25T02:35:17.789144+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:17.787569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:17.787569+0000","last_clean_scrub_stamp":"2024-03-25T02:35:17.787569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:21:38.998744+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2434,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:13.658902+0000","last_change":"2024-03-25T02:35:13.659655+0000","last_active":"2024-03-25T02:35:13.658902+0000","last_peered":"2024-03-25T02:35:13.658902+0000","last_clean":"2024-03-25T02:35:13.658902+0000","last_became_active":"2024-03-25T02:00:33.909846+0000","last_became_peered":"2024-03-25T02:00:33.909846+0000","last_unstale":"2024-03-25T02:35:13.658902+0000","last_undegraded":"2024-03-25T02:35:13.658902+0000","last_fullsized":"2024-03-25T02:35:13.658902+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:13.656196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:13.656196+0000","last_clean_scrub_stamp":"2024-03-25T02:35:13.656196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:57:31.828680+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:17.643919+0000","last_change":"2024-03-25T02:35:17.643919+0000","last_active":"2024-03-25T02:35:17.643919+0000","last_peered":"2024-03-25T02:35:17.643919+0000","last_clean":"2024-03-25T02:35:17.643919+0000","last_became_active":"2024-03-25T02:01:37.856709+0000","last_became_peered":"2024-03-25T02:01:37.856709+0000","last_unstale":"2024-03-25T02:35:17.643919+0000","last_undegraded":"2024-03-25T02:35:17.643919+0000","last_fullsized":"2024-03-25T02:35:17.643919+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:17.641740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:17.641740+0000","last_clean_scrub_stamp":"2024-03-25T02:35:17.641740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:08:21.619756+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2424,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:15.591128+0000","last_change":"2024-03-25T02:35:15.591948+0000","last_active":"2024-03-25T02:35:15.591128+0000","last_peered":"2024-03-25T02:35:15.591128+0000","last_clean":"2024-03-25T02:35:15.591128+0000","last_became_active":"2024-03-25T02:00:33.792626+0000","last_became_peered":"2024-03-25T02:00:33.792626+0000","last_unstale":"2024-03-25T02:35:15.591128+0000","last_undegraded":"2024-03-25T02:35:15.591128+0000","last_fullsized":"2024-03-25T02:35:15.591128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:15.587530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:15.587530+0000","last_clean_scrub_stamp":"2024-03-25T02:35:15.587530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:55:52.995308+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":2421,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:18.574795+0000","last_change":"2024-03-25T02:35:18.574795+0000","last_active":"2024-03-25T02:35:18.574795+0000","last_peered":"2024-03-25T02:35:18.574795+0000","last_clean":"2024-03-25T02:35:18.574795+0000","last_became_active":"2024-03-25T02:01:54.518327+0000","last_became_peered":"2024-03-25T02:01:54.518327+0000","last_unstale":"2024-03-25T02:35:18.574795+0000","last_undegraded":"2024-03-25T02:35:18.574795+0000","last_fullsized":"2024-03-25T02:35:18.574795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:18.572581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:18.572581+0000","last_clean_scrub_stamp":"2024-03-25T02:35:18.572581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T09:05:07.596342+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":2416,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:16.300084+0000","last_change":"2024-03-25T02:35:16.300896+0000","last_active":"2024-03-25T02:35:16.300084+0000","last_peered":"2024-03-25T02:35:16.300084+0000","last_clean":"2024-03-25T02:35:16.300084+0000","last_became_active":"2024-03-25T02:01:53.989515+0000","last_became_peered":"2024-03-25T02:01:53.989515+0000","last_unstale":"2024-03-25T02:35:16.300084+0000","last_undegraded":"2024-03-25T02:35:16.300084+0000","last_fullsized":"2024-03-25T02:35:16.300084+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:16.298538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:16.298538+0000","last_clean_scrub_stamp":"2024-03-25T02:35:16.298538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T02:39:31.877337+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:17.309911+0000","last_change":"2024-03-25T02:35:17.310708+0000","last_active":"2024-03-25T02:35:17.309911+0000","last_peered":"2024-03-25T02:35:17.309911+0000","last_clean":"2024-03-25T02:35:17.309911+0000","last_became_active":"2024-03-25T02:01:55.419583+0000","last_became_peered":"2024-03-25T02:01:55.419583+0000","last_unstale":"2024-03-25T02:35:17.309911+0000","last_undegraded":"2024-03-25T02:35:17.309911+0000","last_fullsized":"2024-03-25T02:35:17.309911+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:17.308338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:17.308338+0000","last_clean_scrub_stamp":"2024-03-25T02:35:17.308338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:30:16.460607+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":2418,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:22.340452+0000","last_change":"2024-03-25T02:35:22.341233+0000","last_active":"2024-03-25T02:35:22.340452+0000","last_peered":"2024-03-25T02:35:22.340452+0000","last_clean":"2024-03-25T02:35:22.340452+0000","last_became_active":"2024-03-25T02:01:56.799197+0000","last_became_peered":"2024-03-25T02:01:56.799197+0000","last_unstale":"2024-03-25T02:35:22.340452+0000","last_undegraded":"2024-03-25T02:35:22.340452+0000","last_fullsized":"2024-03-25T02:35:22.340452+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:22.338871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:22.338871+0000","last_clean_scrub_stamp":"2024-03-25T02:35:22.338871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:03:12.545443+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":2414,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:15.284129+0000","last_change":"2024-03-25T02:35:15.285196+0000","last_active":"2024-03-25T02:35:15.284129+0000","last_peered":"2024-03-25T02:35:15.284129+0000","last_clean":"2024-03-25T02:35:15.284129+0000","last_became_active":"2024-03-25T02:01:50.226409+0000","last_became_peered":"2024-03-25T02:01:50.226409+0000","last_unstale":"2024-03-25T02:35:15.284129+0000","last_undegraded":"2024-03-25T02:35:15.284129+0000","last_fullsized":"2024-03-25T02:35:15.284129+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:15.282456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:15.282456+0000","last_clean_scrub_stamp":"2024-03-25T02:35:15.282456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T11:31:20.828683+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:12.415962+0000","last_change":"2024-03-25T02:35:12.416773+0000","last_active":"2024-03-25T02:35:12.415962+0000","last_peered":"2024-03-25T02:35:12.415962+0000","last_clean":"2024-03-25T02:35:12.415962+0000","last_became_active":"2024-03-25T02:01:39.437348+0000","last_became_peered":"2024-03-25T02:01:39.437348+0000","last_unstale":"2024-03-25T02:35:12.415962+0000","last_undegraded":"2024-03-25T02:35:12.415962+0000","last_fullsized":"2024-03-25T02:35:12.415962+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:12.412308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:12.412308+0000","last_clean_scrub_stamp":"2024-03-25T02:35:12.412308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T14:31:25.033819+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":2426,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:11.699506+0000","last_change":"2024-03-25T02:35:11.700817+0000","last_active":"2024-03-25T02:35:11.699506+0000","last_peered":"2024-03-25T02:35:11.699506+0000","last_clean":"2024-03-25T02:35:11.699506+0000","last_became_active":"2024-03-25T02:01:43.285633+0000","last_became_peered":"2024-03-25T02:01:43.285633+0000","last_unstale":"2024-03-25T02:35:11.699506+0000","last_undegraded":"2024-03-25T02:35:11.699506+0000","last_fullsized":"2024-03-25T02:35:11.699506+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:11.695259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:11.695259+0000","last_clean_scrub_stamp":"2024-03-25T02:35:11.695259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T07:43:52.178518+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":2417,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:13.407144+0000","last_change":"2024-03-25T02:35:13.407902+0000","last_active":"2024-03-25T02:35:13.407144+0000","last_peered":"2024-03-25T02:35:13.407144+0000","last_clean":"2024-03-25T02:35:13.407144+0000","last_became_active":"2024-03-25T02:01:41.631996+0000","last_became_peered":"2024-03-25T02:01:41.631996+0000","last_unstale":"2024-03-25T02:35:13.407144+0000","last_undegraded":"2024-03-25T02:35:13.407144+0000","last_fullsized":"2024-03-25T02:35:13.407144+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:13.405605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:13.405605+0000","last_clean_scrub_stamp":"2024-03-25T02:35:13.405605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:46:42.527510+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":2416,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:19.565941+0000","last_change":"2024-03-25T02:35:19.566700+0000","last_active":"2024-03-25T02:35:19.565941+0000","last_peered":"2024-03-25T02:35:19.565941+0000","last_clean":"2024-03-25T02:35:19.565941+0000","last_became_active":"2024-03-25T02:01:44.123357+0000","last_became_peered":"2024-03-25T02:01:44.123357+0000","last_unstale":"2024-03-25T02:35:19.565941+0000","last_undegraded":"2024-03-25T02:35:19.565941+0000","last_fullsized":"2024-03-25T02:35:19.565941+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:19.562520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:19.562520+0000","last_clean_scrub_stamp":"2024-03-25T02:35:19.562520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T06:05:23.343449+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":2452,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:10.848953+0000","last_change":"2024-03-25T02:35:10.848953+0000","last_active":"2024-03-25T02:35:10.848953+0000","last_peered":"2024-03-25T02:35:10.848953+0000","last_clean":"2024-03-25T02:35:10.848953+0000","last_became_active":"2024-03-25T02:01:40.275307+0000","last_became_peered":"2024-03-25T02:01:40.275307+0000","last_unstale":"2024-03-25T02:35:10.848953+0000","last_undegraded":"2024-03-25T02:35:10.848953+0000","last_fullsized":"2024-03-25T02:35:10.848953+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:10.846551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:10.846551+0000","last_clean_scrub_stamp":"2024-03-25T02:35:10.846551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T12:22:28.639302+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":2425,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:16.847381+0000","last_change":"2024-03-25T02:35:16.848170+0000","last_active":"2024-03-25T02:35:16.847381+0000","last_peered":"2024-03-25T02:35:16.847381+0000","last_clean":"2024-03-25T02:35:16.847381+0000","last_became_active":"2024-03-25T02:01:50.240201+0000","last_became_peered":"2024-03-25T02:01:50.240201+0000","last_unstale":"2024-03-25T02:35:16.847381+0000","last_undegraded":"2024-03-25T02:35:16.847381+0000","last_fullsized":"2024-03-25T02:35:16.847381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:16.845827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:16.845827+0000","last_clean_scrub_stamp":"2024-03-25T02:35:16.845827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T05:57:30.373402+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":2419,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:21.299138+0000","last_change":"2024-03-25T02:35:21.299911+0000","last_active":"2024-03-25T02:35:21.299138+0000","last_peered":"2024-03-25T02:35:21.299138+0000","last_clean":"2024-03-25T02:35:21.299138+0000","last_became_active":"2024-03-25T02:01:56.787332+0000","last_became_peered":"2024-03-25T02:01:56.787332+0000","last_unstale":"2024-03-25T02:35:21.299138+0000","last_undegraded":"2024-03-25T02:35:21.299138+0000","last_fullsized":"2024-03-25T02:35:21.299138+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:21.297510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:21.297510+0000","last_clean_scrub_stamp":"2024-03-25T02:35:21.297510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:08:11.059551+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":2416,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-03-25T02:35:18.598651+0000","last_change":"2024-03-25T02:35:18.598651+0000","last_active":"2024-03-25T02:35:18.598651+0000","last_peered":"2024-03-25T02:35:18.598651+0000","last_clean":"2024-03-25T02:35:18.598651+0000","last_became_active":"2024-03-25T02:01:40.334290+0000","last_became_peered":"2024-03-25T02:01:40.334290+0000","last_unstale":"2024-03-25T02:35:18.598651+0000","last_undegraded":"2024-03-25T02:35:18.598651+0000","last_fullsized":"2024-03-25T02:35:18.598651+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-25T02:35:18.596280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-25T02:35:18.596280+0000","last_clean_scrub_stamp":"2024-03-25T02:35:18.596280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T08:53:33.249195+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":83,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":20,"seq":85899346340,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70924,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300916,"statfs":{"total":96636764160,"available":96564137984,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":20,"seq":85899346346,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71316,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300524,"statfs":{"total":96636764160,"available":96563736576,"internally_reserved":0,"allocated":4141056,"data_stored":1936955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":19,"seq":81604379048,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70928,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300912,"statfs":{"total":96636764160,"available":96564133888,"internally_reserved":0,"allocated":3743744,"data_stored":1539120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379050,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66764,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305076,"statfs":{"total":96636764160,"available":96568397824,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":18,"seq":77309411751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71316,"kb_used_data":4044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300524,"statfs":{"total":96636764160,"available":96563736576,"internally_reserved":0,"allocated":4141056,"data_stored":1936955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509863,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66764,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305076,"statfs":{"total":96636764160,"available":96568397824,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70924,"kb_used_data":3652,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300916,"statfs":{"total":96636764160,"available":96564137984,"internally_reserved":0,"allocated":3739648,"data_stored":1539115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542570,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70928,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300912,"statfs":{"total":96636764160,"available":96564133888,"internally_reserved":0,"allocated":3743744,"data_stored":1539120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-25T02:35:36.661 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-25T02:35:36.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.772+0000 7fe540a60640 10 monclient: get_monmap_and_config 2024-03-25T02:35:36.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.772+0000 7fe540a60640 10 monclient: build_initial_monmap 2024-03-25T02:35:36.761 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.772+0000 7fe540a60640 10 monclient: monmap: 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T02:35:36.774034+0000 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T02:35:36.774034+0000 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:0: v1:172.21.15.83:6789/0 mon.noname-a-legacy 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:1: v1:172.21.15.153:6789/0 mon.noname-c-legacy 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-b 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:3: v2:172.21.15.83:3300/0 mon.noname-a 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr:4: v2:172.21.15.153:3300/0 mon.noname-c 2024-03-25T02:35:36.762 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 1 Processor -- start 2024-03-25T02:35:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 1 -- start start 2024-03-25T02:35:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient: init 2024-03-25T02:35:36.764 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe53c0b0410 addr v1:172.21.15.153:6789/0 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0714d0 0x7fe53c0732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c0714d0 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe53c0737f0 0x7fe53c149d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c0737f0 addr v2:172.21.15.153:3300/0 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe53c072940 con 0x7fe53c0b0410 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fe53c0b07f0 con 0x7fe53c0737f0 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.765 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fe53c071d90 con 0x7fe53c0714d0 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.773+0000 7fe540a60640 10 monclient(hunting): authenticate will time out at 3455.963896s 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe533fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0714d0 0x7fe53c0732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe53ad9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe53c0737f0 0x7fe53c149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe533fff640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0714d0 0x7fe53c0732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:51870/0 (socket says 172.21.15.83:51870) 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe533fff640 1 -- 172.21.15.83:0/3621475413 learned_addr learned my addr 172.21.15.83:0/3621475413 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe533fff640 10 monclient(hunting): get_auth_request con 0x7fe53c0714d0 auth_method 0 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.774+0000 7fe533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.775+0000 7fe533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.775+0000 7fe533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.766 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.775+0000 7fe533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.775+0000 7fe53ad9d640 10 monclient(hunting): get_auth_request con 0x7fe53c0737f0 auth_method 0 2024-03-25T02:35:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.775+0000 7fe53ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.775+0000 7fe53ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.775+0000 7fe53ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.767 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.776+0000 7fe53ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.776+0000 7fe53ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.776+0000 7fe53ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 10 monclient(hunting): handle_auth_done global_id 17303 payload 274 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 1 -- 172.21.15.83:0/3621475413 >> v2:172.21.15.153:3300/0 conn(0x7fe53c0737f0 msgr2=0x7fe53c149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 1 --2- 172.21.15.83:0/3621475413 >> v2:172.21.15.153:3300/0 conn(0x7fe53c0737f0 0x7fe53c149d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 1 -- 172.21.15.83:0/3621475413 >> v1:172.21.15.153:6789/0 conn(0x7fe53c0b0410 legacy=0x7fe53c13c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 10 monclient: _finish_hunting 0 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 1 monclient: found mon.noname-b 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.768 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.777+0000 7fe533fff640 1 -- 172.21.15.83:0/3621475413 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53c072bb0 con 0x7fe53c0714d0 2024-03-25T02:35:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.778+0000 7fe533fff640 1 --2- 172.21.15.83:0/3621475413 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0714d0 0x7fe53c0732b0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7fe52400bb10 tx=0x7fe52402e1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ed98fa43f8c924 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe539d9b640 1 -- 172.21.15.83:0/3621475413 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52403b070 con 0x7fe53c0714d0 2024-03-25T02:35:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe539d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe539d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T02:35:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe539d9b640 10 monclient: dump: 2024-03-25T02:35:36.769 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe539d9b640 10 monclient: _finish_auth 0 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe539d9b640 10 monclient: _check_auth_tickets 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe539d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe540a60640 5 monclient: authenticate success, global_id 17303 2024-03-25T02:35:36.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.780+0000 7fe540a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T02:35:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe539d9b640 1 -- 172.21.15.83:0/3621475413 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe524036440 con 0x7fe53c0714d0 2024-03-25T02:35:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe539d9b640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe539d9b640 1 -- 172.21.15.83:0/3621475413 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524036810 con 0x7fe53c0714d0 2024-03-25T02:35:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe540a60640 10 monclient: get_monmap_and_config success 2024-03-25T02:35:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe540a60640 4 set_mon_vals no callback set 2024-03-25T02:35:36.771 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe539d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe539d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.782+0000 7fe539d9b640 10 monclient: dump: 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.772 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 10 monclient: shutdown 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 -- 172.21.15.83:0/3621475413 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0714d0 msgr2=0x7fe53c0732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 --2- 172.21.15.83:0/3621475413 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0714d0 0x7fe53c0732b0 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7fe52400bb10 tx=0x7fe52402e1f0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 -- 172.21.15.83:0/3621475413 shutdown_connections 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 --2- 172.21.15.83:0/3621475413 >> v2:172.21.15.153:3300/0 conn(0x7fe53c0737f0 0x7fe53c149d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 --2- 172.21.15.83:0/3621475413 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0714d0 0x7fe53c0732b0 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 -- 172.21.15.83:0/3621475413 >> 172.21.15.83:0/3621475413 conn(0x7fe53c0b4ef0 msgr2=0x7fe53c0b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 -- 172.21.15.83:0/3621475413 shutdown_connections 2024-03-25T02:35:36.776 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.783+0000 7fe540a60640 1 -- 172.21.15.83:0/3621475413 wait complete. 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 10 monclient: build_initial_monmap 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 10 monclient: monmap: 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 0 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 0.000000 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:created 0.000000 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 0 (unknown) 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.noname-a 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-b 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.noname-c 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 1 librados: starting msgr at 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 1 librados: starting objecter 2024-03-25T02:35:36.777 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 1 Processor -- start 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 1 -- start start 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 1 librados: setting wanted keys 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 1 librados: calling monclient init 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.784+0000 7fe540a60640 10 monclient: init 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient: _reopen_session rank -1 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe53c0714d0 0x7fe53c0a9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): picked mon.noname-a con 0x7fe53c0714d0 addr [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c0737f0 0x7fe53c13d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): picked mon.noname-b con 0x7fe53c0737f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0b0410 0x7fe53c1d4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): picked mon.noname-c con 0x7fe53c0b0410 addr [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] 2024-03-25T02:35:36.778 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c073c60 con 0x7fe53c0714d0 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe53c071120 con 0x7fe53c0737f0 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): start opening mon connection 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 1 -- --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_getmap magic: 0 -- 0x7fe53c070d80 con 0x7fe53c0b0410 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): _renew_subs 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe540a60640 10 monclient(hunting): authenticate will time out at 3455.975134s 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53ad9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0b0410 0x7fe53c1d4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe53c0714d0 0x7fe53c0a9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53ad9d640 1 --2- >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0b0410 0x7fe53c1d4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3301/0 says I am v2:172.21.15.83:51872/0 (socket says 172.21.15.83:51872) 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53ad9d640 1 -- 172.21.15.83:0/4283027422 learned_addr learned my addr 172.21.15.83:0/4283027422 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53ad9d640 10 monclient(hunting): get_auth_request con 0x7fe53c0b0410 auth_method 0 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.779 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53ad9d640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.785+0000 7fe53ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe533fff640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c0737f0 0x7fe53c13d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53a59c640 10 monclient(hunting): get_auth_request con 0x7fe53c0714d0 auth_method 0 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe533fff640 10 monclient(hunting): get_auth_request con 0x7fe53c0737f0 auth_method 0 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe533fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe533fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.780 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.786+0000 7fe53a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.787+0000 7fe533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-03-25T02:35:36.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.787+0000 7fe533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-03-25T02:35:36.781 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.787+0000 7fe533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 10 monclient(hunting): handle_auth_done global_id 17306 payload 824 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 1 -- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c0737f0 msgr2=0x7fe53c13d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c0737f0 0x7fe53c13d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 1 -- 172.21.15.83:0/4283027422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe53c0714d0 msgr2=0x7fe53c0a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe53c0714d0 0x7fe53c0a9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 10 monclient: _finish_hunting 0 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 1 monclient: found mon.noname-c 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.789+0000 7fe53ad9d640 1 -- 172.21.15.83:0/4283027422 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53c0aab00 con 0x7fe53c0b0410 2024-03-25T02:35:36.782 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.791+0000 7fe53ad9d640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0b0410 0x7fe53c1d4c20 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7fe53400efc0 tx=0x7fe53400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:36.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.791+0000 7fe532ffd640 10 client.?.objecter ms_handle_connect 0x7fe53c0b0410 2024-03-25T02:35:36.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.791+0000 7fe532ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T02:35:36.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534018020 con 0x7fe53c0b0410 2024-03-25T02:35:36.783 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: dump: 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: _finish_auth 0 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: _check_auth_tickets 2024-03-25T02:35:36.784 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe534014480 con 0x7fe53c0b0410 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: handle_config config(0 keys) 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534014870 con 0x7fe53c0b0410 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe532ffd640 10 monclient: dump: 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:epoch 1 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:fsid 86338f78-de3e-4243-afbe-49cfeed24a78 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:last_changed 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:created 2024-03-25T01:58:44.124243+0000 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:min_mon_release 19 (squid) 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:election_strategy: 1 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.b 2024-03-25T02:35:36.785 INFO:teuthology.orchestra.run.smithi083.stderr:2: [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] mon.c 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 5 monclient: authenticate success, global_id 17306 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 10 monclient: _renew_subs 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53c1d4700 con 0x7fe53c0b0410 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe53959a640 4 set_mon_vals no callback set 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 10 client.17306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 10 monclient: _renew_subs 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe53c1d5160 con 0x7fe53c0b0410 2024-03-25T02:35:36.786 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.794+0000 7fe540a60640 1 librados: init done 2024-03-25T02:35:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.795+0000 7fe540a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.795+0000 7fe540a60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.795+0000 7fe540a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:36.787 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:36.795+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe53c0725f0 con 0x7fe53c0b0410 2024-03-25T02:35:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.049+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 4 ==== mgrmap(e 5) ==== 90512+0+0 (secure 0 0 0) 0x7fe534005d10 con 0x7fe53c0b0410 2024-03-25T02:35:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.049+0000 7fe532ffd640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe51c06daa0 0x7fe51c06ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-25T02:35:37.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.050+0000 7fe53a59c640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe51c06daa0 0x7fe51c06ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-25T02:35:37.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.050+0000 7fe53a59c640 10 monclient: get_auth_request con 0x7fe51c06daa0 auth_method 0 2024-03-25T02:35:37.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.050+0000 7fe532ffd640 10 client.17306.objecter ms_handle_connect 0x7fe51c06daa0 2024-03-25T02:35:37.039 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.050+0000 7fe53a59c640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe51c06daa0 0x7fe51c06ff60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0016e0 tx=0x7fe52c009040 comp rx=0 tx=0).ready entity=mgr.4111 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-25T02:35:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 5 ==== osd_map(1250..1250 src has 1060..1250) ==== 6337+0+0 (secure 0 0 0) 0x7fe5340584e0 con 0x7fe53c0b0410 2024-03-25T02:35:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 10 client.17306.objecter ms_dispatch 0x7fe53c000b90 osd_map(1250..1250 src has 1060..1250) 2024-03-25T02:35:37.050 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 3 client.17306.objecter handle_osd_map got epochs [1250,1250] > 0 2024-03-25T02:35:37.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 3 client.17306.objecter handle_osd_map decoding full epoch 1250 2024-03-25T02:35:37.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 20 client.17306.objecter dump_active .. 0 homeless 2024-03-25T02:35:37.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe53c0725f0 con 0x7fe53c0b0410 2024-03-25T02:35:37.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T02:35:37.051 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.061+0000 7fe532ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T02:35:37.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.243+0000 7fe540a60640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-25T02:35:37.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.243+0000 7fe540a60640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-25T02:35:37.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.243+0000 7fe540a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.83:3301/0 2024-03-25T02:35:37.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.243+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 --> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fe53c1d48d0 con 0x7fe53c0b0410 2024-03-25T02:35:37.401 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.412+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe53405a440 con 0x7fe53c0b0410 2024-03-25T02:35:37.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.412+0000 7fe532ffd640 10 monclient: handle_config config(1 keys) 2024-03-25T02:35:37.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.412+0000 7fe53959a640 4 set_mon_vals no callback set 2024-03-25T02:35:37.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.412+0000 7fe53959a640 10 set_mon_vals mon_health_to_clog = false 2024-03-25T02:35:37.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.412+0000 7fe532ffd640 1 -- 172.21.15.83:0/4283027422 <== mon.2 v2:172.21.15.83:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) ==== 123+0+0 (secure 0 0 0) 0x7fe534022070 con 0x7fe53c0b0410 2024-03-25T02:35:37.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.412+0000 7fe532ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-03-25T02:35:37.402 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.412+0000 7fe532ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T02:35:37.404 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.414+0000 7fe540a60640 10 librados: watch_flush enter 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 10 librados: watch_flush exit 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 20 client.17306.objecter shutdown clearing up homeless session... 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 10 client.17306.objecter successfully canceled tick 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe51c06daa0 msgr2=0x7fe51c06ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe51c06daa0 0x7fe51c06ff60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0016e0 tx=0x7fe52c009040 comp rx=0 tx=0).stop 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 10 monclient: shutdown 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0b0410 msgr2=0x7fe53c1d4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0b0410 0x7fe53c1d4c20 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7fe53400efc0 tx=0x7fe53400c490 comp rx=0 tx=0).stop 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 shutdown_connections 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:6800/4072599690,v1:172.21.15.153:6801/4072599690] conn(0x7fe51c06daa0 0x7fe51c06ff60 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.83:3301/0,v1:172.21.15.83:6790/0] conn(0x7fe53c0b0410 0x7fe53c1d4c20 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe53c0737f0 0x7fe53c13d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:37.405 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 --2- 172.21.15.83:0/4283027422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe53c0714d0 0x7fe53c0a9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-25T02:35:37.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 >> 172.21.15.83:0/4283027422 conn(0x7fe53c0b4ef0 msgr2=0x7fe53c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T02:35:37.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 shutdown_connections 2024-03-25T02:35:37.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 -- 172.21.15.83:0/4283027422 wait complete. 2024-03-25T02:35:37.406 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-25T02:35:37.415+0000 7fe540a60640 1 librados: shutdown 2024-03-25T02:35:37.419 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-25T02:35:37.419 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T02:35:37.419 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T02:35:37.419 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:35:37.432 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:35:37.434+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35370) UID: 0 2024-03-25T02:35:37.432 INFO:tasks.ceph.osd.0.smithi083.stderr:2024-03-25T02:35:37.436+0000 ac79640 -1 osd.0 1250 *** Got signal Terminated *** 2024-03-25T02:35:44.050 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:26.898 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.098 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:26.946 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.114 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:26.963 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.148 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:26.997 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.150 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:26.999 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.152 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:26.957 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.153 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:26.958 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.167 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:27.016 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:44.171 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:26.976 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.007 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:31.856 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.008 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:31.857 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.018 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:31.867 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.055 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:31.903 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.056 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:31.905 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.056 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:31.861 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.057 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:31.863 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.065 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:31.914 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.067 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:31.873 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:49.624 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T02:35:49.624 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T02:35:49.624 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:35:49.637 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:35:49.639+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35374) UID: 0 2024-03-25T02:35:49.637 INFO:tasks.ceph.osd.1.smithi083.stderr:2024-03-25T02:35:49.641+0000 ac79640 -1 osd.1 1254 *** Got signal Terminated *** 2024-03-25T02:35:54.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-25T02:35:55.725 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T02:35:55.726 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T02:35:55.726 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:35:55.741 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:35:55.741+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35377) UID: 0 2024-03-25T02:35:55.741 INFO:tasks.ceph.osd.2.smithi083.stderr:2024-03-25T02:35:55.742+0000 ac79640 -1 osd.2 1256 *** Got signal Terminated *** 2024-03-25T02:35:59.060 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:41.909 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.063 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:41.912 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.084 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:41.932 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.112 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:41.917 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.113 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:41.918 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.117 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:41.965 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.119 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:41.968 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.127 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:41.932 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:35:59.139 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:41.987 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:00.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-25T02:36:00.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-25T02:36:01.828 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T02:36:01.828 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T02:36:01.828 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:36:01.841 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:36:01.844+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35381) UID: 0 2024-03-25T02:36:01.841 INFO:tasks.ceph.osd.3.smithi083.stderr:2024-03-25T02:36:01.845+0000 ac79640 -1 osd.3 1258 *** Got signal Terminated *** 2024-03-25T02:36:06.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-25T02:36:09.078 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:51.926 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.084 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:51.933 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.108 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:36:51.956 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.132 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:51.981 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.135 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:51.940 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.137 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:51.942 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.141 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:51.989 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.150 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:36:51.955 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:09.158 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:36:52.006 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.101 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:01.948 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.106 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:01.954 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.116 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:01.965 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.157 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:01.962 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.158 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:01.963 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.169 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:01.973 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.189 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:02.037 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.190 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:02.038 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:19.200 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:02.048 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.646 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:04.494 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.647 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:04.496 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.660 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:04.509 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.696 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:04.500 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.698 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:04.546 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.700 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:04.548 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.702 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:04.506 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.716 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:04.564 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:21.719 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:04.523 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.134 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:11.980 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.136 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:11.985 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.146 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:11.994 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.187 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:11.991 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.188 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:11.993 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.198 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:12.003 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.201 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:12.049 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.206 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:12.054 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:29.216 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:12.064 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.105 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:21.952 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.153 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:22.001 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.163 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:22.011 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.201 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:22.048 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.201 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:22.005 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.202 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:22.050 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.203 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:22.007 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.213 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:22.060 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:39.214 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:22.018 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.173 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:32.020 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.176 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:32.024 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.188 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:32.035 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.227 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:32.031 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.229 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:32.077 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.230 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:32.078 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.231 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:32.035 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.242 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:32.090 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:49.245 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:32.049 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.196 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:42.043 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.199 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:42.047 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.212 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:42.059 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.249 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:42.096 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.250 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:42.097 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.258 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:42.061 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.259 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:42.063 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.264 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:42.111 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:36:59.270 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:42.074 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.223 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:52.070 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.226 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:52.073 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.238 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:37:52.085 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.276 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:52.079 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.277 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:52.080 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.289 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:37:52.092 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.295 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:52.142 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.296 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:52.144 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:09.308 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:37:52.155 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.249 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:02.096 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.250 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:02.098 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.262 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:02.109 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.299 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:02.102 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.300 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:02.104 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.307 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:02.154 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.312 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:02.159 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.312 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:02.116 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:19.323 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:02.170 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.277 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:12.124 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.281 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:12.128 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.293 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:12.140 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.328 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:12.174 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.329 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:12.176 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.329 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:12.132 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.331 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:12.134 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.341 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:12.188 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:29.343 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:12.146 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.298 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:22.144 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.303 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:22.150 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.314 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:22.161 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.350 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:22.153 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.354 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:22.157 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.356 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:22.202 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.358 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:22.205 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.367 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:22.170 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:39.372 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:22.218 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.337 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:37.182 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.340 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:37.186 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.359 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:37.205 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.388 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:37.190 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.389 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:37.192 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.394 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:37.240 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.395 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:37.241 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.404 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:37.206 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:54.409 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:37.255 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.351 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:42.197 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.355 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:42.201 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.364 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:42.210 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.402 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:42.248 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.403 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:42.249 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.405 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:42.207 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.406 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:42.208 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.413 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:42.259 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:37:59.416 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:42.219 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.386 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:57.231 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.390 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:57.236 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.403 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:38:57.249 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.439 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:57.285 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.440 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:57.286 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.444 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:57.246 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.445 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:57.247 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.454 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:38:57.300 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:14.459 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:38:57.261 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.356 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:02.201 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.406 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:02.252 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.420 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:02.266 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.458 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:02.260 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.462 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:02.265 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.470 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:02.316 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.473 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:02.276 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.474 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:02.319 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:19.484 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:02.330 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.363 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T02:38:34.364 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T02:38:34.364 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:38:34.377 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T02:38:34.377+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35356) UID: 0 2024-03-25T02:38:34.378 INFO:tasks.ceph.osd.4.smithi153.stderr:2024-03-25T02:38:34.378+0000 ac79640 -1 osd.4 1260 *** Got signal Terminated *** 2024-03-25T02:38:34.439 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:17.284 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.440 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:17.286 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.456 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:17.302 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.486 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:17.288 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.487 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:17.289 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.502 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:17.304 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.509 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:17.354 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.510 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:17.356 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:34.530 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:17.375 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.404 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:22.249 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.451 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:22.297 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.463 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:22.308 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.499 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:22.344 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.500 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:22.301 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.501 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:22.346 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.501 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:22.303 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.514 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:22.316 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:39.516 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:22.362 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:40.466 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T02:38:40.467 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T02:38:40.467 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:38:40.479 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:38:40.479+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35357) UID: 0 2024-03-25T02:38:40.479 INFO:tasks.ceph.osd.5.smithi153.stderr:2024-03-25T02:38:40.480+0000 ac79640 -1 osd.5 1262 *** Got signal Terminated *** 2024-03-25T02:38:46.569 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T02:38:46.569 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T02:38:46.569 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:38:46.583 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:38:46.582+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35360) UID: 0 2024-03-25T02:38:46.583 INFO:tasks.ceph.osd.6.smithi153.stderr:2024-03-25T02:38:46.583+0000 ac79640 -1 osd.6 1264 *** Got signal Terminated *** 2024-03-25T02:38:54.502 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:37.346 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.502 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:37.346 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.505 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:37.350 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.506 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:37.351 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.523 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:37.368 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.523 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:37.369 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.553 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:37.355 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.554 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:37.356 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:54.571 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:37.372 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.147 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:37.992 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.148 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:37.993 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.158 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:38.003 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.181 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:38.026 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.182 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:38.027 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.193 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:38.038 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.196 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:37.998 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.197 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:37.999 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:38:55.205 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:38.006 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.515 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:57.360 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.565 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:57.410 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.591 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:57.436 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.610 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:57.411 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.611 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:57.412 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.624 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:57.469 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.625 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:57.470 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.628 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:57.429 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:14.642 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:57.487 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.251 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:58.096 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.252 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:58.097 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.260 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:39:58.105 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.261 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:58.062 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.262 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:58.063 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.269 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:39:58.071 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.272 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:58.117 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.273 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:58.118 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:15.281 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:39:58.126 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.548 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:17.392 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.598 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:17.399 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.600 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:17.401 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.613 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:17.457 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.617 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:17.418 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.637 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:17.482 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.655 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:17.499 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.656 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:17.500 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:34.675 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:17.520 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.257 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:18.058 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.258 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:18.059 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.266 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:18.067 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.314 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:18.158 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.315 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:18.159 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.319 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:18.163 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.320 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:18.164 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.325 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:18.169 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:35.329 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:18.173 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.673 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:37.473 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.675 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:37.519 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.677 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:37.477 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.678 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:37.522 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.697 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:37.498 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.698 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:37.542 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.726 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:37.570 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.729 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:37.573 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:54.749 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:37.593 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.249 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:38.049 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.283 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:38.127 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.284 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:38.128 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.292 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:38.136 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.298 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:38.099 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.312 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:38.112 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.313 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:38.157 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.317 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:38.161 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:39:55.326 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:38.170 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.476 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:44.319 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.480 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:44.324 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.496 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:44.340 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.532 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:44.332 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.534 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:44.334 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.536 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:44.380 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.538 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:44.382 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.549 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:44.349 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:01.553 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:44.397 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:11.990 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T02:40:11.990 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T02:40:11.990 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:40:12.003 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:40:12.001+0000 ac79640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35361) UID: 0 2024-03-25T02:40:12.005 INFO:tasks.ceph.osd.7.smithi153.stderr:2024-03-25T02:40:12.002+0000 ac79640 -1 osd.7 1264 *** Got signal Terminated *** 2024-03-25T02:40:14.729 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:57.572 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.731 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:57.575 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.747 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:57.591 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.778 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:57.578 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.778 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:57.622 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.782 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:57.583 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.783 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:57.626 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.796 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:57.640 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:14.801 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:57.601 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.391 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:58.191 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.392 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:58.193 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.403 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:40:58.203 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.409 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:58.252 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.409 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:58.253 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.410 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:58.254 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.411 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:58.255 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.422 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:40:58.264 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:15.422 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:40:58.265 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.778 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:17.621 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.782 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:17.626 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.801 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:17.645 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.831 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:17.674 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.832 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:17.632 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.834 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:17.677 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.834 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:17.634 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.852 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:17.695 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:34.859 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:17.659 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.445 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:18.244 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.447 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:18.245 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.454 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:18.253 34746== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.457 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:18.299 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.457 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:18.299 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.458 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:18.300 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.458 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:18.300 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.465 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:18.308 35072== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:35.466 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:18.308 35077== Warning: unimplemented fcntl command: 1036 2024-03-25T02:40:41.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-25T02:40:42.499 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T02:40:42.499 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T02:40:42.499 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T02:40:42.499 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:40:42.525 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T02:40:42.525 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T02:40:42.525 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:40:42.526 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T02:40:42.531+0000 7f6ff4346640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34744) UID: 0 2024-03-25T02:40:42.526 INFO:tasks.ceph.mgr.x.smithi153.stderr:2024-03-25T02:40:42.531+0000 7f6ff4346640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T02:40:42.556 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T02:40:42.556 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T02:40:42.556 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T02:40:42.556 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:40:42.580 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-25T02:40:42.576+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35042) UID: 0 2024-03-25T02:40:42.581 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-03-25T02:40:42.578+0000 9f99640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T02:40:43.410 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:26.253 35072== 2024-03-25T02:40:43.410 INFO:tasks.ceph.mon.a.smithi083.stderr:==00:00:41:26.253 35072== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T02:40:43.432 INFO:tasks.ceph.mon.a.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T02:40:46.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-25T02:40:46.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-25T02:40:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T02:40:48.658 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T02:40:48.659 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T02:40:48.660 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T02:40:48.660 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:40:48.678 INFO:tasks.ceph.mon.c.smithi083.stderr:2024-03-25T02:40:48.671+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35044) UID: 0 2024-03-25T02:40:48.679 INFO:tasks.ceph.mon.c.smithi083.stderr:2024-03-25T02:40:48.672+0000 9f99640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T02:40:49.221 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:32.064 35077== 2024-03-25T02:40:49.221 INFO:tasks.ceph.mon.c.smithi083.stderr:==00:00:41:32.064 35077== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T02:40:49.239 INFO:tasks.ceph.mon.c.smithi083.stderr:daemon-helper: command failed with exit status 42 2024-03-25T02:40:51.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-25T02:40:51.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-25T02:40:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T02:40:54.762 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T02:40:54.762 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T02:40:54.763 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T02:40:54.763 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T02:40:54.777 INFO:tasks.ceph.mon.b.smithi153.stderr:2024-03-25T02:40:54.773+0000 9f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34742) UID: 0 2024-03-25T02:40:54.779 INFO:tasks.ceph.mon.b.smithi153.stderr:2024-03-25T02:40:54.774+0000 9f99640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T02:40:55.370 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:38.169 34746== 2024-03-25T02:40:55.370 INFO:tasks.ceph.mon.b.smithi153.stderr:==00:00:41:38.169 34746== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T02:40:55.390 INFO:tasks.ceph.mon.b.smithi153.stderr:daemon-helper: command failed with exit status 42 2024-03-25T02:40:56.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-25T02:40:56.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-25T02:41:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T02:41:00.864 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T02:41:00.865 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T02:41:00.865 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T02:41:00.865 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T02:41:00.907 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T02:41:00.907 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T02:41:01.090 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T02:41:01.090 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T02:41:01.162 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T02:41:01.162 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T02:41:01.229 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T02:41:01.229 DEBUG:teuthology.orchestra.run.smithi083:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T02:41:01.293 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T02:41:01.294 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T02:41:01.396 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T02:41:01.397 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T02:41:01.480 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T02:41:01.480 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T02:41:01.552 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T02:41:01.552 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T02:41:01.611 INFO:tasks.ceph:Archiving mon data... 2024-03-25T02:41:01.613 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/data/mon.a.tgz 2024-03-25T02:41:01.613 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-25T02:41:01.631 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.in8fXjwNOc 2024-03-25T02:41:01.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.in8fXjwNOc 2024-03-25T02:41:01.796 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.in8fXjwNOc 2024-03-25T02:41:01.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-25T02:41:01.883 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.in8fXjwNOc is 487KB 2024-03-25T02:41:01.981 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.in8fXjwNOc 2024-03-25T02:41:01.996 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/data/mon.c.tgz 2024-03-25T02:41:01.997 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-03-25T02:41:02.050 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.cPAhBl6yz4 2024-03-25T02:41:02.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.cPAhBl6yz4 2024-03-25T02:41:02.200 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.cPAhBl6yz4 2024-03-25T02:41:02.284 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.cPAhBl6yz4 is 489KB 2024-03-25T02:41:02.382 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.cPAhBl6yz4 2024-03-25T02:41:02.396 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/data/mon.b.tgz 2024-03-25T02:41:02.397 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-25T02:41:02.415 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.5ajLPDsRHC 2024-03-25T02:41:02.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.5ajLPDsRHC 2024-03-25T02:41:02.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.5ajLPDsRHC 2024-03-25T02:41:02.674 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.5ajLPDsRHC is 489KB 2024-03-25T02:41:02.776 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.5ajLPDsRHC 2024-03-25T02:41:02.791 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T02:41:02.791 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T02:41:02.793 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T02:41:03.142 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T02:41:03.413 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T02:41:03.416 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/remote/smithi083/crash 2024-03-25T02:41:03.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T02:41:03.444 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/remote/smithi153/crash 2024-03-25T02:41:03.445 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T02:41:03.471 INFO:tasks.ceph:Compressing logs... 2024-03-25T02:41:03.472 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T02:41:03.489 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T02:41:03.569 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T02:41:03.569 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-03-25T02:41:03.569 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T02:41:03.569 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35190.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T02:41:03.569 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T02:41:03.569 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-03-25T02:41:03.570 INFO:teuthology.orchestra.run.smithi153.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T02:41:03.570 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T02:41:03.570 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T02:41:03.570 INFO:teuthology.orchestra.run.smithi153.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T02:41:03.570 INFO:teuthology.orchestra.run.smithi153.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T02:41:03.570 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34899.log 2024-03-25T02:41:03.570 INFO:teuthology.orchestra.run.smithi153.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T02:41:03.571 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35019.log 2024-03-25T02:41:03.571 INFO:teuthology.orchestra.run.smithi153.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T02:41:03.571 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35059.log 2024-03-25T02:41:03.571 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35019.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T02:41:03.571 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35059.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34899.log.gz 2024-03-25T02:41:03.571 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T02:41:03.571 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35019.log.gz 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35059.log.gz 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34939.log 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35150.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34939.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T02:41:03.572 INFO:teuthology.orchestra.run.smithi153.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mgr.x.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34939.log.gz 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34979.log 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.34979.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34979.log.gz 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35099.log 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.b.log: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.35099.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35099.log.gz 2024-03-25T02:41:03.573 INFO:teuthology.orchestra.run.smithi153.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59507.log 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108924.log 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40098.log 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39767.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108924.log.gz 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40098.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39768.log 2024-03-25T02:41:03.607 INFO:teuthology.orchestra.run.smithi083.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40343.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40098.log.gz 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.59507.log.gz 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52417.log 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39768.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gz 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108840.log 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52417.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39768.log.gz 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108840.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108840.log.gz 2024-03-25T02:41:03.608 INFO:teuthology.orchestra.run.smithi083.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.1.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39381.log 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41448.log 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.52417.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38859.log 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41448.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39381.log.gz 2024-03-25T02:41:03.609 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.38859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40023.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35535.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8%/var/log/ceph/ceph-client.admin.40103.log: -- replaced with /var/log/ceph/ceph-client.admin.41448.log.gz 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38859.log.gz 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108378.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40710.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35535.log: /var/log/ceph/ceph-client.admin.108378.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40023.log.gz 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52419.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35535.log.gz 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43087.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108378.log.gz 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108420.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52419.log: /var/log/ceph/ceph-client.admin.43087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43200.log 2024-03-25T02:41:03.610 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40385.log 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43088.log 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108420.log.gz 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39117.log 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43088.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.43200.log.gz 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40385.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41049.log 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39117.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.52419.log.gz 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.43088.log.gz 2024-03-25T02:41:03.611 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39117.log.gz 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52418.log 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41049.log: /var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108084.log 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.52418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59510.log 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108084.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.43087.log.gz 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:gzip 87.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.108504.log 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108084.log.gz 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41205.log 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39687.log 2024-03-25T02:41:03.612 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.52418.log.gz 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108504.log.gz 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41273.log 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39944.log 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41273.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.59510.log.gz 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66993.log 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41049.log.gz 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41066.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39944.log.gz 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40860.log 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40719.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.66993.log.gz 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108756.log 2024-03-25T02:41:03.613 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40860.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41205.log.gz 2024-03-25T02:41:03.614 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41325.log 2024-03-25T02:41:03.614 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108756.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108756.log.gz 2024-03-25T02:41:03.614 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41679.log 2024-03-25T02:41:03.614 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz/var/log/ceph/ceph-client.admin.41325.log: 2024-03-25T02:41:03.614 INFO:teuthology.orchestra.run.smithi083.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gz 2024-03-25T02:41:03.614 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-03-25T02:41:03.614 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41786.log 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39390.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41273.log.gz 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.40860.log.gz 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr: 89.3% -- /var/log/ceph/ceph-client.admin.59509.log 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41679.log.gz 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41786.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-03-25T02:41:03.615 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39661.log 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42908.log 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108964.log 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42908.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39661.log.gz 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41245.log 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108964.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.59509.log.gz 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39389.log 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108964.log.gz 2024-03-25T02:41:03.616 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59508.log 2024-03-25T02:41:03.617 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39389.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39389.log.gz 2024-03-25T02:41:03.617 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-03-25T02:41:03.617 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59508.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.42908.log.gz 2024-03-25T02:41:03.617 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40301.log 2024-03-25T02:41:03.617 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40770.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-03-25T02:41:03.617 INFO:teuthology.orchestra.run.smithi083.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.59508.log.gz 2024-03-25T02:41:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37508.log 2024-03-25T02:41:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39615.log 2024-03-25T02:41:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.37508.log: /var/log/ceph/ceph-client.admin.40301.log: 88.2% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40301.log.gz 2024-03-25T02:41:03.618 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37508.log.gz 2024-03-25T02:41:03.618 INFO:teuthology.orchestra.run.smithi083.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41245.log.gz 2024-03-25T02:41:03.618 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108630.log 2024-03-25T02:41:03.619 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40247.log 2024-03-25T02:41:03.619 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66992.log 2024-03-25T02:41:03.619 INFO:teuthology.orchestra.run.smithi083.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39615.log.gz 2024-03-25T02:41:03.619 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40247.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108588.log 2024-03-25T02:41:03.619 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108630.log.gz 2024-03-25T02:41:03.619 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-03-25T02:41:03.619 INFO:teuthology.orchestra.run.smithi083.stderr: 90.3%/var/log/ceph/ceph-client.admin.108588.log: -- replaced with /var/log/ceph/ceph-client.admin.41786.log.gz 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40247.log.gz 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41368.log 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108588.log.gz 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40837.log: /var/log/ceph/ceph-client.admin.41368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.1.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.66992.log.gz 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T02:41:03.620 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41857.log 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41857.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.41368.log.gz 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41747.log 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108126.log 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41747.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41857.log.gz 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42907.log 2024-03-25T02:41:03.621 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108126.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108126.log.gz 2024-03-25T02:41:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-25T02:41:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42907.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.42907.log.gz 2024-03-25T02:41:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-03-25T02:41:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40790.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-03-25T02:41:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29276.log 2024-03-25T02:41:03.622 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39388.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-03-25T02:41:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-25T02:41:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.tmp-client.admin.29276.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29276.log.gz 2024-03-25T02:41:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41605.log 2024-03-25T02:41:03.623 INFO:teuthology.orchestra.run.smithi083.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.40710.log.gz/var/log/ceph/ceph-client.admin.40003.log: 2024-03-25T02:41:03.623 INFO:teuthology.orchestra.run.smithi083.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-03-25T02:41:03.623 INFO:teuthology.orchestra.run.smithi083.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-25T02:41:03.623 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39623.log 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36380.log 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108210.log 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.36380.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41605.log.gz 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39623.log.gz 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 88.0% 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36380.log.gz 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108210.log.gz 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38909.log 2024-03-25T02:41:03.624 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35386.log 2024-03-25T02:41:03.625 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.38909.log: /var/log/ceph/ceph-client.admin.40741.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38909.log.gz 2024-03-25T02:41:03.625 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40217.log 2024-03-25T02:41:03.625 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35386.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35386.log.gz 2024-03-25T02:41:03.625 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59506.log 2024-03-25T02:41:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40217.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40217.log.gz 2024-03-25T02:41:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108672.log 2024-03-25T02:41:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.59506.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.59506.log.gz 2024-03-25T02:41:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38657.log 2024-03-25T02:41:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108672.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108672.log.gz 2024-03-25T02:41:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41427.log 2024-03-25T02:41:03.626 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.38657.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38657.log.gz 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41534.log 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41488.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41427.log.gz 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41747.log.gz 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35413.log 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108294.log 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39382.log 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108294.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35413.log.gz 2024-03-25T02:41:03.627 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2024-03-25T02:41:03.628 INFO:teuthology.orchestra.run.smithi083.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108294.log.gz 2024-03-25T02:41:03.628 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43199.log 2024-03-25T02:41:03.628 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41190.log: 87.5% 94.1% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-03-25T02:41:03.628 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2024-03-25T02:41:03.628 INFO:teuthology.orchestra.run.smithi083.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41534.log.gz 2024-03-25T02:41:03.628 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108546.log 2024-03-25T02:41:03.628 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41536.log 2024-03-25T02:41:03.629 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108546.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T02:41:03.629 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41536.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108546.log.gz 2024-03-25T02:41:03.629 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108252.log 2024-03-25T02:41:03.643 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mgr.y.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.41536.log.gz 2024-03-25T02:41:03.643 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108798.log 2024-03-25T02:41:03.643 INFO:teuthology.orchestra.run.smithi083.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.43199.log.gz 2024-03-25T02:41:03.643 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108252.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2024-03-25T02:41:03.643 INFO:teuthology.orchestra.run.smithi083.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108252.log.gz 2024-03-25T02:41:03.643 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41815.log 2024-03-25T02:41:03.643 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40845.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108798.log.gz 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40845.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr: 95.0% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41776.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40944.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40123.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gz 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35084.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40123.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40123.log.gz 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42909.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.35084.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.35084.log.gz 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41210.log 2024-03-25T02:41:03.644 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.42909.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42909.log.gz 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39666.log 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41210.log: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39666.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39666.log.gz 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108168.log 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39923.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108168.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108168.log.gz 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41020.log 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.0.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-03-25T02:41:03.645 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108336.log 2024-03-25T02:41:03.648 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41585.log 2024-03-25T02:41:03.654 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108336.log: 92.2% 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41020.log.gz 2024-03-25T02:41:03.654 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41776.log.gz 2024-03-25T02:41:03.655 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108042.log 2024-03-25T02:41:03.655 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41594.log 2024-03-25T02:41:03.655 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108042.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108042.log.gz 2024-03-25T02:41:03.655 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T02:41:03.657 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41594.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41594.log.gz 2024-03-25T02:41:03.657 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41896.log 2024-03-25T02:41:03.657 INFO:teuthology.orchestra.run.smithi083.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41585.log.gz 2024-03-25T02:41:03.657 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41828.log 2024-03-25T02:41:03.662 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41896.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108336.log.gz 2024-03-25T02:41:03.665 INFO:teuthology.orchestra.run.smithi083.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.41896.log.gz 2024-03-25T02:41:03.665 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-03-25T02:41:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41828.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41828.log.gz 2024-03-25T02:41:03.669 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log 2024-03-25T02:41:03.671 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40966.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T02:41:03.672 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108714.log 2024-03-25T02:41:03.676 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41890.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gz 2024-03-25T02:41:03.676 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T02:41:03.677 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108714.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108714.log.gz 2024-03-25T02:41:03.677 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41610.log 2024-03-25T02:41:03.690 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39209.log 2024-03-25T02:41:03.710 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43089.log 2024-03-25T02:41:03.717 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39209.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41322.log 2024-03-25T02:41:03.717 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39209.log.gz 2024-03-25T02:41:03.732 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.43089.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.43089.log.gz 2024-03-25T02:41:03.735 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2024-03-25T02:41:03.736 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41322.log: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.41610.log.gz 2024-03-25T02:41:03.748 INFO:teuthology.orchestra.run.smithi083.stderr: 92.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66991.log 2024-03-25T02:41:03.749 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39165.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2024-03-25T02:41:03.749 INFO:teuthology.orchestra.run.smithi083.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-03-25T02:41:03.759 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108882.log 2024-03-25T02:41:03.774 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.66991.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.66991.log.gz 2024-03-25T02:41:03.775 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-03-25T02:41:03.777 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108882.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.108882.log.gz 2024-03-25T02:41:03.793 INFO:teuthology.orchestra.run.smithi083.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.41322.log.gz 2024-03-25T02:41:03.794 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39647.log 2024-03-25T02:41:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41871.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-03-25T02:41:03.807 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108462.log 2024-03-25T02:41:03.808 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39647.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39647.log.gz 2024-03-25T02:41:03.808 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-03-25T02:41:03.810 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.108462.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108462.log.gz 2024-03-25T02:41:03.810 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39385.log 2024-03-25T02:41:03.815 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41125.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-03-25T02:41:03.815 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41591.log 2024-03-25T02:41:03.816 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39385.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39385.log.gz 2024-03-25T02:41:03.816 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40690.log 2024-03-25T02:41:03.820 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.41591.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41591.log.gz 2024-03-25T02:41:03.820 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38955.log 2024-03-25T02:41:03.834 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.40690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-03-25T02:41:03.835 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.38955.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-03-25T02:41:03.843 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T02:41:03.845 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-client.admin.39075.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-03-25T02:41:03.911 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/ceph-mon.c.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.40690.log.gz 2024-03-25T02:41:04.018 INFO:teuthology.orchestra.run.smithi153.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T02:41:04.156 INFO:teuthology.orchestra.run.smithi083.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.40944.log.gz 2024-03-25T02:41:04.977 INFO:teuthology.orchestra.run.smithi083.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40845.log.gz 2024-03-25T02:41:05.798 INFO:teuthology.orchestra.run.smithi083.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T02:41:06.087 INFO:teuthology.orchestra.run.smithi153.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T02:41:06.520 INFO:teuthology.orchestra.run.smithi083.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T02:41:09.538 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T02:41:09.597 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T02:41:10.017 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T02:41:10.459 INFO:teuthology.orchestra.run.smithi083.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T02:41:10.583 INFO:teuthology.orchestra.run.smithi083.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T02:41:10.658 INFO:teuthology.orchestra.run.smithi083.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T02:41:11.114 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T02:41:11.116 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-25T02:41:11.116 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m7.585s 2024-03-25T02:41:11.116 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m28.102s 2024-03-25T02:41:11.116 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.989s 2024-03-25T02:41:11.135 INFO:teuthology.orchestra.run.smithi083.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T02:41:11.137 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-25T02:41:11.137 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m7.637s 2024-03-25T02:41:11.137 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m34.837s 2024-03-25T02:41:11.137 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m1.200s 2024-03-25T02:41:11.137 INFO:tasks.ceph:Archiving logs... 2024-03-25T02:41:11.138 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/remote/smithi083/log 2024-03-25T02:41:11.139 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T02:41:16.269 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/remote/smithi153/log 2024-03-25T02:41:16.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T02:41:21.662 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T02:41:21.663 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T02:41:21.840 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T02:41:21.841 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T02:41:21.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T02:41:21.844 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T02:41:21.922 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T02:41:22.020 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T02:41:22.020 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T02:41:22.027 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T02:41:22.042 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-25T02:41:22.045 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-25T02:41:22.047 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T02:41:22.047 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-25T02:41:22.047 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 34 -287us[ -287us] +/- 70ms 2024-03-25T02:41:22.047 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 +618us[ +623us] +/- 58ms 2024-03-25T02:41:22.047 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -497us[ -497us] +/- 79ms 2024-03-25T02:41:22.048 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T02:41:22.049 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T02:41:22.050 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-25T02:41:22.050 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 -293us[ -293us] +/- 70ms 2024-03-25T02:41:22.050 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 +617us[ +612us] +/- 58ms 2024-03-25T02:41:22.050 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -495us[ -495us] +/- 79ms 2024-03-25T02:41:22.050 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T02:41:22.051 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T02:41:22.078 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T02:41:22.079 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T02:41:22.090 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T02:41:22.163 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T02:41:22.237 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-25T02:41:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T02:41:22.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-25T02:41:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T02:41:22.307 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T02:41:22.317 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T02:41:22.344 INFO:teuthology.task.internal:Duration was 3465.649172 seconds 2024-03-25T02:41:22.345 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T02:41:22.354 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T02:41:22.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T02:41:22.357 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T02:41:22.398 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T02:41:22.401 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T02:41:22.767 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T02:41:22.768 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-03-25T02:41:22.768 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T02:41:22.830 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-25T02:41:22.830 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T02:41:22.855 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T02:41:22.855 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T02:41:22.873 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T02:41:22.925 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T02:41:22.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T02:41:22.944 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T02:41:23.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T02:41:23.076 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T02:41:23.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T02:41:23.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T02:41:23.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T02:41:23.142 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T02:41:23.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T02:41:23.168 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-03-25T02:41:23.196 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-25T02:41:23.217 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T02:41:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T02:41:23.255 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T02:41:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T02:41:23.272 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T02:41:23.284 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T02:41:23.285 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/remote/smithi083 2024-03-25T02:41:23.285 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T02:41:23.352 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770/remote/smithi153 2024-03-25T02:41:23.352 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T02:41:23.388 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T02:41:23.388 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T02:41:23.392 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T02:41:23.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T02:41:23.448 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T02:41:23.448 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T02:41:23.457 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T02:41:23.457 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T02:41:23.460 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T02:41:23.471 INFO:teuthology.orchestra.run.smithi083.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 02:41 /home/ubuntu/cephtest 2024-03-25T02:41:23.492 INFO:teuthology.orchestra.run.smithi153.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 02:41 /home/ubuntu/cephtest 2024-03-25T02:41:23.493 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T02:41:23.503 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T02:41:23.515 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T02:41:23.516 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T02:41:23.540 DEBUG:teuthology.nuke:shortname: smithi083 2024-03-25T02:41:23.540 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T02:41:23.561 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-25T02:41:23.561 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T02:41:23.565 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 01:34:33.974788', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBObskKB0hMJpfUxtnW8i01ugVMaSIKTiGMPE9pwaWD74qdcrRNDF80UoV9ZtlOqvva7+TeHF3DSRg//B/c0wnVw='} 2024-03-25T02:41:23.585 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620770', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 01:34:33.972861', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCSw63IetpNrST60r4TDQMhJcu7ayY4c4BA+U3CukOW/GwnEviD1/7HU+6Ttzt5Qh8lFWze4ACmLVo6Q1ugJG2k='} 2024-03-25T02:41:23.614 INFO:teuthology.orchestra.console.smithi083:Power off 2024-03-25T02:41:23.614 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T02:41:23.641 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-03-25T02:41:23.642 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T02:41:23.659 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-25T02:41:23.659 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T02:41:23.672 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-25T02:41:23.686 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-25T02:41:23.686 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T02:41:23.713 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-25T02:41:27.673 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T02:41:27.715 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T02:41:27.784 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-25T02:41:27.825 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-25T02:41:31.786 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T02:41:31.826 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T02:41:31.896 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-03-25T02:41:31.937 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-25T02:41:31.997 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-03-25T02:41:32.038 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-25T02:41:32.143 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 3465.6491718292236 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-03-25T02:41:32.143 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T02:41:32.220 INFO:teuthology.run:FAIL